Note: This is test shard 3 of 8.
[==========] Running 7 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from IgnoredTserverGoesDownDuringRebalancingTest
[ RUN ] IgnoredTserverGoesDownDuringRebalancingTest.TserverDown
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240501 14:08:48.133080 7326 test_util.cc:268] Using random seed: 108733379
I20240501 14:08:48.139900 7326 ts_itest-base.cc:112] Starting cluster with:
I20240501 14:08:48.140014 7326 ts_itest-base.cc:113] --------------
I20240501 14:08:48.140049 7326 ts_itest-base.cc:114] 8 tablet servers
I20240501 14:08:48.140074 7326 ts_itest-base.cc:115] 3 replicas per TS
I20240501 14:08:48.140089 7326 ts_itest-base.cc:116] --------------
2024-05-01T14:08:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T14:08:48Z Disabled control of system clock
I20240501 14:08:48.149713 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.39.190:44225
--webserver_interface=127.7.39.190
--webserver_port=0
--builtin_ntp_servers=127.7.39.148:46693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.39.190:44225
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240501 14:08:48.254701 7338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:48.254981 7338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:48.255023 7338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:48.258029 7338 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:08:48.258106 7338 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 14:08:48.258129 7338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:48.258147 7338 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 14:08:48.258163 7338 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 14:08:48.261664 7338 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:46693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.7.39.190:44225
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.39.190:44225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.7.39.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.7338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:48.262529 7338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:48.263687 7338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:48.271687 7346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:48.273015 7344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:48.274237 7343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:48.275449 7338 server_base.cc:1034] running on GCE node
I20240501 14:08:48.275974 7338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:48.276921 7338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:48.278065 7338 hybrid_clock.cc:648] HybridClock initialized: now 1714572528278060 us; error 30 us; skew 500 ppm
I20240501 14:08:48.284363 7338 webserver.cc:460] Webserver started at http://127.7.39.190:46585/ using document root <none> and password file <none>
I20240501 14:08:48.285010 7338 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:48.285077 7338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:48.285264 7338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:48.286998 7338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "d04daab32424495099974a7bc137239f"
format_stamp: "Formatted at 2024-05-01 14:08:48 on dist-test-slave-rnfw"
I20240501 14:08:48.287510 7338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "d04daab32424495099974a7bc137239f"
format_stamp: "Formatted at 2024-05-01 14:08:48 on dist-test-slave-rnfw"
I20240501 14:08:48.290980 7338 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:08:48.293287 7352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:48.294270 7338 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:48.294396 7338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/wal
uuid: "d04daab32424495099974a7bc137239f"
format_stamp: "Formatted at 2024-05-01 14:08:48 on dist-test-slave-rnfw"
I20240501 14:08:48.294482 7338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:48.307150 7338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:48.307768 7338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:48.307940 7338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:48.314256 7338 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.190:44225
I20240501 14:08:48.314254 7404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.190:44225 every 8 connection(s)
I20240501 14:08:48.315282 7338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 14:08:48.317800 7405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:48.322674 7405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d04daab32424495099974a7bc137239f: Bootstrap starting.
I20240501 14:08:48.324929 7405 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d04daab32424495099974a7bc137239f: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:48.325664 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 7338
I20240501 14:08:48.325764 7405 log.cc:824] T 00000000000000000000000000000000 P d04daab32424495099974a7bc137239f: Log is configured to *not* fsync() on all Append() calls
I20240501 14:08:48.325810 7326 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 14:08:48.327797 7405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d04daab32424495099974a7bc137239f: No bootstrap required, opened a new log
I20240501 14:08:48.330904 7405 raft_consensus.cc:348] T 00000000000000000000000000000000 P d04daab32424495099974a7bc137239f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04daab32424495099974a7bc137239f" member_type: VOTER last_known_addr { host: "127.7.39.190" port: 44225 } }
I20240501 14:08:48.331115 7405 raft_consensus.cc:374] T 00000000000000000000000000000000 P d04daab32424495099974a7bc137239f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:48.331177 7405 raft_consensus.cc:729] T 00000000000000000000000000000000 P d04daab32424495099974a7bc137239f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d04daab32424495099974a7bc137239f, State: Initialized, Role: FOLLOWER
I20240501 14:08:48.331565 7405 consensus_queue.cc:260] T 00000000000000000000000000000000 P d04daab32424495099974a7bc137239f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04daab32424495099974a7bc137239f" member_type: VOTER last_known_addr { host: "127.7.39.190" port: 44225 } }
I20240501 14:08:48.331684 7405 raft_consensus.cc:388] T 00000000000000000000000000000000 P d04daab32424495099974a7bc137239f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:08:48.331750 7405 raft_consensus.cc:482] T 00000000000000000000000000000000 P d04daab32424495099974a7bc137239f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:08:48.331856 7405 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d04daab32424495099974a7bc137239f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:48.332672 7405 raft_consensus.cc:504] T 00000000000000000000000000000000 P d04daab32424495099974a7bc137239f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04daab32424495099974a7bc137239f" member_type: VOTER last_known_addr { host: "127.7.39.190" port: 44225 } }
I20240501 14:08:48.333021 7405 leader_election.cc:302] T 00000000000000000000000000000000 P d04daab32424495099974a7bc137239f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d04daab32424495099974a7bc137239f; no voters:
I20240501 14:08:48.333344 7405 leader_election.cc:288] T 00000000000000000000000000000000 P d04daab32424495099974a7bc137239f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:08:48.333457 7410 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d04daab32424495099974a7bc137239f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:48.333750 7410 raft_consensus.cc:686] T 00000000000000000000000000000000 P d04daab32424495099974a7bc137239f [term 1 LEADER]: Becoming Leader. State: Replica: d04daab32424495099974a7bc137239f, State: Running, Role: LEADER
I20240501 14:08:48.334105 7410 consensus_queue.cc:237] T 00000000000000000000000000000000 P d04daab32424495099974a7bc137239f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04daab32424495099974a7bc137239f" member_type: VOTER last_known_addr { host: "127.7.39.190" port: 44225 } }
I20240501 14:08:48.334434 7405 sys_catalog.cc:564] T 00000000000000000000000000000000 P d04daab32424495099974a7bc137239f [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:08:48.335312 7410 sys_catalog.cc:455] T 00000000000000000000000000000000 P d04daab32424495099974a7bc137239f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d04daab32424495099974a7bc137239f. Latest consensus state: current_term: 1 leader_uuid: "d04daab32424495099974a7bc137239f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04daab32424495099974a7bc137239f" member_type: VOTER last_known_addr { host: "127.7.39.190" port: 44225 } } }
I20240501 14:08:48.335510 7410 sys_catalog.cc:458] T 00000000000000000000000000000000 P d04daab32424495099974a7bc137239f [sys.catalog]: This master's current role is: LEADER
I20240501 14:08:48.335680 7411 sys_catalog.cc:455] T 00000000000000000000000000000000 P d04daab32424495099974a7bc137239f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d04daab32424495099974a7bc137239f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04daab32424495099974a7bc137239f" member_type: VOTER last_known_addr { host: "127.7.39.190" port: 44225 } } }
I20240501 14:08:48.335858 7411 sys_catalog.cc:458] T 00000000000000000000000000000000 P d04daab32424495099974a7bc137239f [sys.catalog]: This master's current role is: LEADER
I20240501 14:08:48.335904 7419 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:08:48.338994 7419 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:08:48.344641 7419 catalog_manager.cc:1347] Generated new cluster ID: d3ec98fcf17149f5a1c15b2979f0042b
I20240501 14:08:48.344724 7419 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:08:48.356725 7419 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 14:08:48.357945 7419 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:08:48.366804 7419 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d04daab32424495099974a7bc137239f: Generated new TSK 0
I20240501 14:08:48.367650 7419 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:08:48.372799 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.129:0
--local_ip_for_outbound_sockets=127.7.39.129
--webserver_interface=127.7.39.129
--webserver_port=0
--tserver_master_addrs=127.7.39.190:44225
--builtin_ntp_servers=127.7.39.148:46693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240501 14:08:48.480765 7429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:48.480994 7429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:48.481045 7429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:48.483985 7429 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:08:48.484052 7429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:48.484118 7429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.129
I20240501 14:08:48.487612 7429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:46693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.7.39.129
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.7.39.190:44225
--never_fsync=true
--heap_profile_path=/tmp/kudu.7429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.129
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:48.488531 7429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:48.489647 7429 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:48.499074 7434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:48.500121 7435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:48.501258 7437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:48.502452 7429 server_base.cc:1034] running on GCE node
I20240501 14:08:48.502849 7429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:48.503456 7429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:48.504611 7429 hybrid_clock.cc:648] HybridClock initialized: now 1714572528504595 us; error 29 us; skew 500 ppm
I20240501 14:08:48.506522 7429 webserver.cc:460] Webserver started at http://127.7.39.129:42839/ using document root <none> and password file <none>
I20240501 14:08:48.507081 7429 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:48.507153 7429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:48.507316 7429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:48.512475 7429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "ae52f888f5094416a24cb3dffbff03c4"
format_stamp: "Formatted at 2024-05-01 14:08:48 on dist-test-slave-rnfw"
I20240501 14:08:48.512984 7429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "ae52f888f5094416a24cb3dffbff03c4"
format_stamp: "Formatted at 2024-05-01 14:08:48 on dist-test-slave-rnfw"
I20240501 14:08:48.516455 7429 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:08:48.518549 7443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:48.519557 7429 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:08:48.519649 7429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "ae52f888f5094416a24cb3dffbff03c4"
format_stamp: "Formatted at 2024-05-01 14:08:48 on dist-test-slave-rnfw"
I20240501 14:08:48.519721 7429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:48.532742 7429 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:48.533355 7429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:48.533495 7429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:48.533995 7429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:48.534930 7429 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:48.534984 7429 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:48.535059 7429 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:48.535094 7429 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:48.543819 7429 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.129:37083
I20240501 14:08:48.543857 7556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.129:37083 every 8 connection(s)
I20240501 14:08:48.544755 7429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 14:08:48.551647 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 7429
I20240501 14:08:48.551762 7326 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 14:08:48.553184 7557 heartbeater.cc:347] Connected to a master server at 127.7.39.190:44225
I20240501 14:08:48.553462 7557 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:48.554013 7557 heartbeater.cc:511] Master 127.7.39.190:44225 requested a full tablet report, sending...
I20240501 14:08:48.554065 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.130:0
--local_ip_for_outbound_sockets=127.7.39.130
--webserver_interface=127.7.39.130
--webserver_port=0
--tserver_master_addrs=127.7.39.190:44225
--builtin_ntp_servers=127.7.39.148:46693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 14:08:48.555622 7368 ts_manager.cc:194] Registered new tserver with Master: ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083)
I20240501 14:08:48.556813 7368 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.129:34055
W20240501 14:08:48.662035 7560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:48.662277 7560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:48.662326 7560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:48.665464 7560 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:08:48.665549 7560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:48.665632 7560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.130
I20240501 14:08:48.669133 7560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:46693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.7.39.130
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.7.39.190:44225
--never_fsync=true
--heap_profile_path=/tmp/kudu.7560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.130
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:48.670032 7560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:48.671128 7560 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:48.680413 7568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:48.681630 7566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:48.682696 7565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:48.683882 7560 server_base.cc:1034] running on GCE node
I20240501 14:08:48.684330 7560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:48.684935 7560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:48.686098 7560 hybrid_clock.cc:648] HybridClock initialized: now 1714572528686075 us; error 41 us; skew 500 ppm
I20240501 14:08:48.688122 7560 webserver.cc:460] Webserver started at http://127.7.39.130:38779/ using document root <none> and password file <none>
I20240501 14:08:48.688657 7560 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:48.688710 7560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:48.688874 7560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:48.690495 7560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "9e44efa72e7d4421b284ae74d1cb9634"
format_stamp: "Formatted at 2024-05-01 14:08:48 on dist-test-slave-rnfw"
I20240501 14:08:48.690923 7560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "9e44efa72e7d4421b284ae74d1cb9634"
format_stamp: "Formatted at 2024-05-01 14:08:48 on dist-test-slave-rnfw"
I20240501 14:08:48.694202 7560 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20240501 14:08:48.696547 7574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:48.697542 7560 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:48.697649 7560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "9e44efa72e7d4421b284ae74d1cb9634"
format_stamp: "Formatted at 2024-05-01 14:08:48 on dist-test-slave-rnfw"
I20240501 14:08:48.697757 7560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:48.724050 7560 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:48.724705 7560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:48.724879 7560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:48.725422 7560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:48.726343 7560 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:48.726393 7560 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:48.726452 7560 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:48.726478 7560 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:48.735458 7560 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.130:46433
I20240501 14:08:48.735491 7687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.130:46433 every 8 connection(s)
I20240501 14:08:48.736443 7560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 14:08:48.741436 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 7560
I20240501 14:08:48.741551 7326 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 14:08:48.743831 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.131:0
--local_ip_for_outbound_sockets=127.7.39.131
--webserver_interface=127.7.39.131
--webserver_port=0
--tserver_master_addrs=127.7.39.190:44225
--builtin_ntp_servers=127.7.39.148:46693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 14:08:48.745584 7688 heartbeater.cc:347] Connected to a master server at 127.7.39.190:44225
I20240501 14:08:48.745815 7688 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:48.746315 7688 heartbeater.cc:511] Master 127.7.39.190:44225 requested a full tablet report, sending...
I20240501 14:08:48.747236 7368 ts_manager.cc:194] Registered new tserver with Master: 9e44efa72e7d4421b284ae74d1cb9634 (127.7.39.130:46433)
I20240501 14:08:48.747792 7368 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.130:41387
W20240501 14:08:48.852995 7691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:48.853230 7691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:48.853294 7691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:48.856402 7691 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:08:48.856477 7691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:48.856571 7691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.131
I20240501 14:08:48.860276 7691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:46693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.7.39.131
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.7.39.190:44225
--never_fsync=true
--heap_profile_path=/tmp/kudu.7691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.131
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:48.861196 7691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:48.862319 7691 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:48.871754 7699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:48.872673 7696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:48.874264 7697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:48.875382 7691 server_base.cc:1034] running on GCE node
I20240501 14:08:48.875874 7691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:48.876506 7691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:48.877660 7691 hybrid_clock.cc:648] HybridClock initialized: now 1714572528877642 us; error 42 us; skew 500 ppm
I20240501 14:08:48.879971 7691 webserver.cc:460] Webserver started at http://127.7.39.131:33587/ using document root <none> and password file <none>
I20240501 14:08:48.880635 7691 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:48.880708 7691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:48.880946 7691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:48.882714 7691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "9ae5a87ca07b40a788a42734b24f66c6"
format_stamp: "Formatted at 2024-05-01 14:08:48 on dist-test-slave-rnfw"
I20240501 14:08:48.883237 7691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "9ae5a87ca07b40a788a42734b24f66c6"
format_stamp: "Formatted at 2024-05-01 14:08:48 on dist-test-slave-rnfw"
I20240501 14:08:48.886788 7691 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20240501 14:08:48.889739 7705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:48.890846 7691 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240501 14:08:48.890950 7691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "9ae5a87ca07b40a788a42734b24f66c6"
format_stamp: "Formatted at 2024-05-01 14:08:48 on dist-test-slave-rnfw"
I20240501 14:08:48.891158 7691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:48.907677 7691 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:48.908282 7691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:48.908469 7691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:48.909046 7691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:48.910015 7691 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:48.910060 7691 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:48.910102 7691 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:48.910116 7691 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:48.918839 7691 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.131:34697
I20240501 14:08:48.918902 7818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.131:34697 every 8 connection(s)
I20240501 14:08:48.919828 7691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 14:08:48.920403 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 7691
I20240501 14:08:48.920507 7326 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 14:08:48.922830 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.132:0
--local_ip_for_outbound_sockets=127.7.39.132
--webserver_interface=127.7.39.132
--webserver_port=0
--tserver_master_addrs=127.7.39.190:44225
--builtin_ntp_servers=127.7.39.148:46693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 14:08:48.933991 7819 heartbeater.cc:347] Connected to a master server at 127.7.39.190:44225
I20240501 14:08:48.934309 7819 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:48.934943 7819 heartbeater.cc:511] Master 127.7.39.190:44225 requested a full tablet report, sending...
I20240501 14:08:48.936020 7368 ts_manager.cc:194] Registered new tserver with Master: 9ae5a87ca07b40a788a42734b24f66c6 (127.7.39.131:34697)
I20240501 14:08:48.936611 7368 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.131:58805
W20240501 14:08:49.107435 7822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:49.107829 7822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:49.107956 7822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:49.114125 7822 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:08:49.114257 7822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:49.114451 7822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.132
I20240501 14:08:49.118691 7822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:46693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.7.39.132
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.7.39.190:44225
--never_fsync=true
--heap_profile_path=/tmp/kudu.7822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.132
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:49.119724 7822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:49.120904 7822 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:49.131287 7827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:49.132470 7828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:49.133494 7830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:49.134920 7822 server_base.cc:1034] running on GCE node
I20240501 14:08:49.135422 7822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:49.136102 7822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:49.137265 7822 hybrid_clock.cc:648] HybridClock initialized: now 1714572529137208 us; error 84 us; skew 500 ppm
I20240501 14:08:49.139570 7822 webserver.cc:460] Webserver started at http://127.7.39.132:33291/ using document root <none> and password file <none>
I20240501 14:08:49.140146 7822 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:49.140216 7822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:49.140465 7822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:49.142328 7822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "47eabc135e9f4237b036097085e729d8"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-rnfw"
I20240501 14:08:49.142832 7822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "47eabc135e9f4237b036097085e729d8"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-rnfw"
I20240501 14:08:49.146353 7822 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240501 14:08:49.148662 7836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:49.149684 7822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:49.149789 7822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "47eabc135e9f4237b036097085e729d8"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-rnfw"
I20240501 14:08:49.149894 7822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:49.174737 7822 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:49.175451 7822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:49.175622 7822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:49.176183 7822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:49.177071 7822 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:49.177119 7822 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:49.177177 7822 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:49.177209 7822 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:49.185937 7822 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.132:42369
I20240501 14:08:49.185957 7949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.132:42369 every 8 connection(s)
I20240501 14:08:49.186890 7822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 14:08:49.191309 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 7822
I20240501 14:08:49.191401 7326 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 14:08:49.193337 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.133:0
--local_ip_for_outbound_sockets=127.7.39.133
--webserver_interface=127.7.39.133
--webserver_port=0
--tserver_master_addrs=127.7.39.190:44225
--builtin_ntp_servers=127.7.39.148:46693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 14:08:49.195817 7950 heartbeater.cc:347] Connected to a master server at 127.7.39.190:44225
I20240501 14:08:49.196048 7950 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:49.196581 7950 heartbeater.cc:511] Master 127.7.39.190:44225 requested a full tablet report, sending...
I20240501 14:08:49.197616 7368 ts_manager.cc:194] Registered new tserver with Master: 47eabc135e9f4237b036097085e729d8 (127.7.39.132:42369)
I20240501 14:08:49.198437 7368 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.132:41455
W20240501 14:08:49.301182 7953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:49.301416 7953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:49.301463 7953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:49.304481 7953 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:08:49.304553 7953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:49.304621 7953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.133
I20240501 14:08:49.308254 7953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:46693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.7.39.133
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.7.39.190:44225
--never_fsync=true
--heap_profile_path=/tmp/kudu.7953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.133
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:49.309127 7953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:49.310222 7953 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:49.319641 7958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:49.320870 7959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:49.321936 7961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:49.323201 7953 server_base.cc:1034] running on GCE node
I20240501 14:08:49.323663 7953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:49.324285 7953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:49.325462 7953 hybrid_clock.cc:648] HybridClock initialized: now 1714572529325453 us; error 29 us; skew 500 ppm
I20240501 14:08:49.327536 7953 webserver.cc:460] Webserver started at http://127.7.39.133:33043/ using document root <none> and password file <none>
I20240501 14:08:49.328096 7953 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:49.328151 7953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:49.328320 7953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:49.329972 7953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "9faf312806f54f2e8a4a807b234e0d1b"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-rnfw"
I20240501 14:08:49.330433 7953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "9faf312806f54f2e8a4a807b234e0d1b"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-rnfw"
I20240501 14:08:49.333758 7953 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 14:08:49.336033 7967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:49.337004 7953 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:08:49.337106 7953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "9faf312806f54f2e8a4a807b234e0d1b"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-rnfw"
I20240501 14:08:49.337188 7953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:49.346652 7953 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:49.347287 7953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:49.347465 7953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:49.347991 7953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:49.348887 7953 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:49.348943 7953 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:49.349005 7953 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:49.349041 7953 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:49.358152 7953 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.133:38805
I20240501 14:08:49.358172 8080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.133:38805 every 8 connection(s)
I20240501 14:08:49.359177 7953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 14:08:49.359606 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 7953
I20240501 14:08:49.359742 7326 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 14:08:49.362112 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.134:0
--local_ip_for_outbound_sockets=127.7.39.134
--webserver_interface=127.7.39.134
--webserver_port=0
--tserver_master_addrs=127.7.39.190:44225
--builtin_ntp_servers=127.7.39.148:46693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 14:08:49.368232 8081 heartbeater.cc:347] Connected to a master server at 127.7.39.190:44225
I20240501 14:08:49.368503 8081 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:49.369226 8081 heartbeater.cc:511] Master 127.7.39.190:44225 requested a full tablet report, sending...
I20240501 14:08:49.370376 7368 ts_manager.cc:194] Registered new tserver with Master: 9faf312806f54f2e8a4a807b234e0d1b (127.7.39.133:38805)
I20240501 14:08:49.370922 7368 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.133:55453
W20240501 14:08:49.471072 8084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:49.471303 8084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:49.471352 8084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:49.474309 8084 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:08:49.474413 8084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:49.474500 8084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.134
I20240501 14:08:49.477998 8084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:46693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.7.39.134
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.7.39.190:44225
--never_fsync=true
--heap_profile_path=/tmp/kudu.8084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.134
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:49.478886 8084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:49.480010 8084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:49.489012 8090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:49.490170 8092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:49.491166 8089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:49.492429 8084 server_base.cc:1034] running on GCE node
I20240501 14:08:49.492862 8084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:49.493429 8084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:49.494575 8084 hybrid_clock.cc:648] HybridClock initialized: now 1714572529494565 us; error 33 us; skew 500 ppm
I20240501 14:08:49.496651 8084 webserver.cc:460] Webserver started at http://127.7.39.134:38807/ using document root <none> and password file <none>
I20240501 14:08:49.497215 8084 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:49.497272 8084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:49.497478 8084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:49.499236 8084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "9d7c8c0f7907428eab22633702ead7ab"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-rnfw"
I20240501 14:08:49.499701 8084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "9d7c8c0f7907428eab22633702ead7ab"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-rnfw"
I20240501 14:08:49.503032 8084 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:08:49.505179 8098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:49.506136 8084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240501 14:08:49.506247 8084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "9d7c8c0f7907428eab22633702ead7ab"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-rnfw"
I20240501 14:08:49.506333 8084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:49.520661 8084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:49.521260 8084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:49.521417 8084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:49.521931 8084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:49.522794 8084 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:49.522835 8084 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:49.522876 8084 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:49.522924 8084 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:49.531293 8084 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.134:38699
I20240501 14:08:49.531324 8211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.134:38699 every 8 connection(s)
I20240501 14:08:49.532239 8084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 14:08:49.538378 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 8084
I20240501 14:08:49.538509 7326 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 14:08:49.540735 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.135:0
--local_ip_for_outbound_sockets=127.7.39.135
--webserver_interface=127.7.39.135
--webserver_port=0
--tserver_master_addrs=127.7.39.190:44225
--builtin_ntp_servers=127.7.39.148:46693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 14:08:49.541105 8212 heartbeater.cc:347] Connected to a master server at 127.7.39.190:44225
I20240501 14:08:49.541321 8212 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:49.541924 8212 heartbeater.cc:511] Master 127.7.39.190:44225 requested a full tablet report, sending...
I20240501 14:08:49.543255 7368 ts_manager.cc:194] Registered new tserver with Master: 9d7c8c0f7907428eab22633702ead7ab (127.7.39.134:38699)
I20240501 14:08:49.543890 7368 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.134:57327
I20240501 14:08:49.559512 7557 heartbeater.cc:503] Master 127.7.39.190:44225 was elected leader, sending a full tablet report...
W20240501 14:08:49.647820 8215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:49.648056 8215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:49.648133 8215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:49.651110 8215 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:08:49.651176 8215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:49.651244 8215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.135
I20240501 14:08:49.654815 8215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:46693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.7.39.135
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.7.39.190:44225
--never_fsync=true
--heap_profile_path=/tmp/kudu.8215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.135
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:49.655728 8215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:49.656782 8215 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:49.665829 8221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:49.667181 8220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:49.668128 8223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:49.669404 8215 server_base.cc:1034] running on GCE node
I20240501 14:08:49.669814 8215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:49.670382 8215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:49.671537 8215 hybrid_clock.cc:648] HybridClock initialized: now 1714572529671512 us; error 45 us; skew 500 ppm
I20240501 14:08:49.673503 8215 webserver.cc:460] Webserver started at http://127.7.39.135:35631/ using document root <none> and password file <none>
I20240501 14:08:49.674052 8215 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:49.674131 8215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:49.674336 8215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:49.676079 8215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "b4105a6d531d401faadf67b894fec79d"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-rnfw"
I20240501 14:08:49.676519 8215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "b4105a6d531d401faadf67b894fec79d"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-rnfw"
I20240501 14:08:49.679796 8215 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:08:49.681854 8229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:49.682833 8215 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:49.682937 8215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "b4105a6d531d401faadf67b894fec79d"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-rnfw"
I20240501 14:08:49.683025 8215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:49.699218 8215 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:49.699859 8215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:49.700035 8215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:49.700577 8215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:49.701558 8215 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:49.701614 8215 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:49.701679 8215 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:49.701710 8215 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:49.711141 8215 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.135:39611
I20240501 14:08:49.711170 8342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.135:39611 every 8 connection(s)
I20240501 14:08:49.712086 8215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 14:08:49.717826 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 8215
I20240501 14:08:49.717940 7326 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 14:08:49.720216 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.136:0
--local_ip_for_outbound_sockets=127.7.39.136
--webserver_interface=127.7.39.136
--webserver_port=0
--tserver_master_addrs=127.7.39.190:44225
--builtin_ntp_servers=127.7.39.148:46693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 14:08:49.720906 8343 heartbeater.cc:347] Connected to a master server at 127.7.39.190:44225
I20240501 14:08:49.721160 8343 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:49.721810 8343 heartbeater.cc:511] Master 127.7.39.190:44225 requested a full tablet report, sending...
I20240501 14:08:49.722914 7368 ts_manager.cc:194] Registered new tserver with Master: b4105a6d531d401faadf67b894fec79d (127.7.39.135:39611)
I20240501 14:08:49.723775 7368 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.135:39237
I20240501 14:08:49.749651 7688 heartbeater.cc:503] Master 127.7.39.190:44225 was elected leader, sending a full tablet report...
W20240501 14:08:49.834723 8346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:49.834977 8346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:49.835103 8346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:49.838264 8346 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:08:49.838354 8346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:49.838454 8346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.136
I20240501 14:08:49.842088 8346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:46693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.7.39.136
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.7.39.190:44225
--never_fsync=true
--heap_profile_path=/tmp/kudu.8346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.136
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:49.842989 8346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:49.844168 8346 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:49.853382 8352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:49.854749 8351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:49.855751 8354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:49.857087 8346 server_base.cc:1034] running on GCE node
I20240501 14:08:49.857489 8346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:49.858038 8346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:49.859200 8346 hybrid_clock.cc:648] HybridClock initialized: now 1714572529859188 us; error 32 us; skew 500 ppm
I20240501 14:08:49.861215 8346 webserver.cc:460] Webserver started at http://127.7.39.136:45159/ using document root <none> and password file <none>
I20240501 14:08:49.861779 8346 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:49.861840 8346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:49.862028 8346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:49.863713 8346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "9d32e843754e44d0b9aae6f472a4d9a2"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-rnfw"
I20240501 14:08:49.864187 8346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "9d32e843754e44d0b9aae6f472a4d9a2"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-rnfw"
I20240501 14:08:49.867614 8346 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:08:49.869822 8360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:49.870785 8346 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:08:49.870883 8346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "9d32e843754e44d0b9aae6f472a4d9a2"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-rnfw"
I20240501 14:08:49.870978 8346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:49.886992 8346 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:49.887773 8346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:49.887952 8346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:49.888484 8346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:49.889369 8346 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:49.889428 8346 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:49.889473 8346 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:49.889498 8346 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:49.898952 8346 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.136:37475
I20240501 14:08:49.899001 8473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.136:37475 every 8 connection(s)
I20240501 14:08:49.900146 8346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 14:08:49.907227 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 8346
I20240501 14:08:49.907363 7326 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 14:08:49.909194 8474 heartbeater.cc:347] Connected to a master server at 127.7.39.190:44225
I20240501 14:08:49.909411 8474 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:49.910074 8474 heartbeater.cc:511] Master 127.7.39.190:44225 requested a full tablet report, sending...
I20240501 14:08:49.911010 7368 ts_manager.cc:194] Registered new tserver with Master: 9d32e843754e44d0b9aae6f472a4d9a2 (127.7.39.136:37475)
I20240501 14:08:49.911538 7368 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.136:53675
I20240501 14:08:49.922015 7326 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 14:08:49.934150 7326 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 14:08:49.934247 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 7953
I20240501 14:08:49.938676 7819 heartbeater.cc:503] Master 127.7.39.190:44225 was elected leader, sending a full tablet report...
I20240501 14:08:49.940412 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 8084
I20240501 14:08:49.947903 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 8215
I20240501 14:08:49.953190 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 8346
I20240501 14:08:50.200417 7950 heartbeater.cc:503] Master 127.7.39.190:44225 was elected leader, sending a full tablet report...
I20240501 14:08:53.710898 7368 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32870:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:08:53.731015 7490 tablet_service.cc:1467] Processing CreateTablet for tablet 2cf2f85819584fdf872f6c590d11f873 (DEFAULT_TABLE table=rebalance_test_table_0 [id=54950e0e732146d89f1bde2556d9c1da]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:53.731031 7753 tablet_service.cc:1467] Processing CreateTablet for tablet ceee52d679bf4d8e8896a6c0dde0519b (DEFAULT_TABLE table=rebalance_test_table_0 [id=54950e0e732146d89f1bde2556d9c1da]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:53.731366 7883 tablet_service.cc:1467] Processing CreateTablet for tablet 14e325d8d3184c9192b8cebafb414190 (DEFAULT_TABLE table=rebalance_test_table_0 [id=54950e0e732146d89f1bde2556d9c1da]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:53.732048 7490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cf2f85819584fdf872f6c590d11f873. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:53.732048 7753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceee52d679bf4d8e8896a6c0dde0519b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:53.732335 7883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14e325d8d3184c9192b8cebafb414190. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:53.734280 7752 tablet_service.cc:1467] Processing CreateTablet for tablet 2cf2f85819584fdf872f6c590d11f873 (DEFAULT_TABLE table=rebalance_test_table_0 [id=54950e0e732146d89f1bde2556d9c1da]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:53.734586 7752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cf2f85819584fdf872f6c590d11f873. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:53.735204 7491 tablet_service.cc:1467] Processing CreateTablet for tablet ceee52d679bf4d8e8896a6c0dde0519b (DEFAULT_TABLE table=rebalance_test_table_0 [id=54950e0e732146d89f1bde2556d9c1da]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:53.735497 7491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceee52d679bf4d8e8896a6c0dde0519b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:53.739249 7884 tablet_service.cc:1467] Processing CreateTablet for tablet ceee52d679bf4d8e8896a6c0dde0519b (DEFAULT_TABLE table=rebalance_test_table_0 [id=54950e0e732146d89f1bde2556d9c1da]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:53.739567 7884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceee52d679bf4d8e8896a6c0dde0519b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:53.740115 7489 tablet_service.cc:1467] Processing CreateTablet for tablet 14e325d8d3184c9192b8cebafb414190 (DEFAULT_TABLE table=rebalance_test_table_0 [id=54950e0e732146d89f1bde2556d9c1da]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:53.740417 7489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14e325d8d3184c9192b8cebafb414190. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:53.741982 8495 tablet_bootstrap.cc:492] T 2cf2f85819584fdf872f6c590d11f873 P 9ae5a87ca07b40a788a42734b24f66c6: Bootstrap starting.
I20240501 14:08:53.743153 8494 tablet_bootstrap.cc:492] T ceee52d679bf4d8e8896a6c0dde0519b P ae52f888f5094416a24cb3dffbff03c4: Bootstrap starting.
I20240501 14:08:53.745232 8495 tablet_bootstrap.cc:654] T 2cf2f85819584fdf872f6c590d11f873 P 9ae5a87ca07b40a788a42734b24f66c6: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:53.745779 8496 tablet_bootstrap.cc:492] T ceee52d679bf4d8e8896a6c0dde0519b P 47eabc135e9f4237b036097085e729d8: Bootstrap starting.
I20240501 14:08:53.746224 8495 log.cc:824] T 2cf2f85819584fdf872f6c590d11f873 P 9ae5a87ca07b40a788a42734b24f66c6: Log is configured to *not* fsync() on all Append() calls
I20240501 14:08:53.747123 8494 tablet_bootstrap.cc:654] T ceee52d679bf4d8e8896a6c0dde0519b P ae52f888f5094416a24cb3dffbff03c4: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:53.747824 8496 tablet_bootstrap.cc:654] T ceee52d679bf4d8e8896a6c0dde0519b P 47eabc135e9f4237b036097085e729d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:53.748384 8494 log.cc:824] T ceee52d679bf4d8e8896a6c0dde0519b P ae52f888f5094416a24cb3dffbff03c4: Log is configured to *not* fsync() on all Append() calls
I20240501 14:08:53.748641 8496 log.cc:824] T ceee52d679bf4d8e8896a6c0dde0519b P 47eabc135e9f4237b036097085e729d8: Log is configured to *not* fsync() on all Append() calls
I20240501 14:08:53.749032 8495 tablet_bootstrap.cc:492] T 2cf2f85819584fdf872f6c590d11f873 P 9ae5a87ca07b40a788a42734b24f66c6: No bootstrap required, opened a new log
I20240501 14:08:53.749226 8495 ts_tablet_manager.cc:1367] T 2cf2f85819584fdf872f6c590d11f873 P 9ae5a87ca07b40a788a42734b24f66c6: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240501 14:08:53.750414 8496 tablet_bootstrap.cc:492] T ceee52d679bf4d8e8896a6c0dde0519b P 47eabc135e9f4237b036097085e729d8: No bootstrap required, opened a new log
I20240501 14:08:53.750602 8496 ts_tablet_manager.cc:1367] T ceee52d679bf4d8e8896a6c0dde0519b P 47eabc135e9f4237b036097085e729d8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 14:08:53.750424 7621 tablet_service.cc:1467] Processing CreateTablet for tablet 14e325d8d3184c9192b8cebafb414190 (DEFAULT_TABLE table=rebalance_test_table_0 [id=54950e0e732146d89f1bde2556d9c1da]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:53.751468 7621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14e325d8d3184c9192b8cebafb414190. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:53.751570 8494 tablet_bootstrap.cc:492] T ceee52d679bf4d8e8896a6c0dde0519b P ae52f888f5094416a24cb3dffbff03c4: No bootstrap required, opened a new log
I20240501 14:08:53.751770 8494 ts_tablet_manager.cc:1367] T ceee52d679bf4d8e8896a6c0dde0519b P ae52f888f5094416a24cb3dffbff03c4: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.002s
I20240501 14:08:53.750427 7622 tablet_service.cc:1467] Processing CreateTablet for tablet 2cf2f85819584fdf872f6c590d11f873 (DEFAULT_TABLE table=rebalance_test_table_0 [id=54950e0e732146d89f1bde2556d9c1da]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:53.753208 7622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cf2f85819584fdf872f6c590d11f873. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:53.753312 8495 raft_consensus.cc:348] T 2cf2f85819584fdf872f6c590d11f873 P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:08:53.753501 8495 raft_consensus.cc:374] T 2cf2f85819584fdf872f6c590d11f873 P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:53.753552 8495 raft_consensus.cc:729] T 2cf2f85819584fdf872f6c590d11f873 P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ae5a87ca07b40a788a42734b24f66c6, State: Initialized, Role: FOLLOWER
I20240501 14:08:53.753973 8495 consensus_queue.cc:260] T 2cf2f85819584fdf872f6c590d11f873 P 9ae5a87ca07b40a788a42734b24f66c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:08:53.754917 8495 ts_tablet_manager.cc:1398] T 2cf2f85819584fdf872f6c590d11f873 P 9ae5a87ca07b40a788a42734b24f66c6: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240501 14:08:53.754935 8496 raft_consensus.cc:348] T ceee52d679bf4d8e8896a6c0dde0519b P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:08:53.755147 8494 raft_consensus.cc:348] T ceee52d679bf4d8e8896a6c0dde0519b P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:08:53.755291 8495 tablet_bootstrap.cc:492] T ceee52d679bf4d8e8896a6c0dde0519b P 9ae5a87ca07b40a788a42734b24f66c6: Bootstrap starting.
I20240501 14:08:53.755321 8494 raft_consensus.cc:374] T ceee52d679bf4d8e8896a6c0dde0519b P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:53.755369 8494 raft_consensus.cc:729] T ceee52d679bf4d8e8896a6c0dde0519b P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae52f888f5094416a24cb3dffbff03c4, State: Initialized, Role: FOLLOWER
I20240501 14:08:53.755810 8494 consensus_queue.cc:260] T ceee52d679bf4d8e8896a6c0dde0519b P ae52f888f5094416a24cb3dffbff03c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:08:53.756193 8495 tablet_bootstrap.cc:654] T ceee52d679bf4d8e8896a6c0dde0519b P 9ae5a87ca07b40a788a42734b24f66c6: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:53.756644 8494 ts_tablet_manager.cc:1398] T ceee52d679bf4d8e8896a6c0dde0519b P ae52f888f5094416a24cb3dffbff03c4: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240501 14:08:53.758011 8494 tablet_bootstrap.cc:492] T 2cf2f85819584fdf872f6c590d11f873 P ae52f888f5094416a24cb3dffbff03c4: Bootstrap starting.
I20240501 14:08:53.759079 8494 tablet_bootstrap.cc:654] T 2cf2f85819584fdf872f6c590d11f873 P ae52f888f5094416a24cb3dffbff03c4: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:53.759702 8496 raft_consensus.cc:374] T ceee52d679bf4d8e8896a6c0dde0519b P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:53.759774 8496 raft_consensus.cc:729] T ceee52d679bf4d8e8896a6c0dde0519b P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47eabc135e9f4237b036097085e729d8, State: Initialized, Role: FOLLOWER
I20240501 14:08:53.760231 8496 consensus_queue.cc:260] T ceee52d679bf4d8e8896a6c0dde0519b P 47eabc135e9f4237b036097085e729d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:08:53.761070 8496 ts_tablet_manager.cc:1398] T ceee52d679bf4d8e8896a6c0dde0519b P 47eabc135e9f4237b036097085e729d8: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20240501 14:08:53.761200 8494 tablet_bootstrap.cc:492] T 2cf2f85819584fdf872f6c590d11f873 P ae52f888f5094416a24cb3dffbff03c4: No bootstrap required, opened a new log
I20240501 14:08:53.761299 8495 tablet_bootstrap.cc:492] T ceee52d679bf4d8e8896a6c0dde0519b P 9ae5a87ca07b40a788a42734b24f66c6: No bootstrap required, opened a new log
I20240501 14:08:53.761389 8495 ts_tablet_manager.cc:1367] T ceee52d679bf4d8e8896a6c0dde0519b P 9ae5a87ca07b40a788a42734b24f66c6: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:08:53.761332 8494 ts_tablet_manager.cc:1367] T 2cf2f85819584fdf872f6c590d11f873 P ae52f888f5094416a24cb3dffbff03c4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:08:53.761669 8504 tablet_bootstrap.cc:492] T 2cf2f85819584fdf872f6c590d11f873 P 9e44efa72e7d4421b284ae74d1cb9634: Bootstrap starting.
I20240501 14:08:53.761914 8495 raft_consensus.cc:348] T ceee52d679bf4d8e8896a6c0dde0519b P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:08:53.762028 8495 raft_consensus.cc:374] T ceee52d679bf4d8e8896a6c0dde0519b P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:53.762071 8495 raft_consensus.cc:729] T ceee52d679bf4d8e8896a6c0dde0519b P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ae5a87ca07b40a788a42734b24f66c6, State: Initialized, Role: FOLLOWER
I20240501 14:08:53.762209 8496 tablet_bootstrap.cc:492] T 14e325d8d3184c9192b8cebafb414190 P 47eabc135e9f4237b036097085e729d8: Bootstrap starting.
I20240501 14:08:53.762228 8495 consensus_queue.cc:260] T ceee52d679bf4d8e8896a6c0dde0519b P 9ae5a87ca07b40a788a42734b24f66c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:08:53.762163 8494 raft_consensus.cc:348] T 2cf2f85819584fdf872f6c590d11f873 P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:08:53.762423 8494 raft_consensus.cc:374] T 2cf2f85819584fdf872f6c590d11f873 P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:53.762513 8494 raft_consensus.cc:729] T 2cf2f85819584fdf872f6c590d11f873 P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae52f888f5094416a24cb3dffbff03c4, State: Initialized, Role: FOLLOWER
I20240501 14:08:53.762629 8495 ts_tablet_manager.cc:1398] T ceee52d679bf4d8e8896a6c0dde0519b P 9ae5a87ca07b40a788a42734b24f66c6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:53.762792 8494 consensus_queue.cc:260] T 2cf2f85819584fdf872f6c590d11f873 P ae52f888f5094416a24cb3dffbff03c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:08:53.763257 8496 tablet_bootstrap.cc:654] T 14e325d8d3184c9192b8cebafb414190 P 47eabc135e9f4237b036097085e729d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:53.763418 8494 ts_tablet_manager.cc:1398] T 2cf2f85819584fdf872f6c590d11f873 P ae52f888f5094416a24cb3dffbff03c4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:53.763657 8494 tablet_bootstrap.cc:492] T 14e325d8d3184c9192b8cebafb414190 P ae52f888f5094416a24cb3dffbff03c4: Bootstrap starting.
I20240501 14:08:53.764662 8494 tablet_bootstrap.cc:654] T 14e325d8d3184c9192b8cebafb414190 P ae52f888f5094416a24cb3dffbff03c4: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:53.765364 8504 tablet_bootstrap.cc:654] T 2cf2f85819584fdf872f6c590d11f873 P 9e44efa72e7d4421b284ae74d1cb9634: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:53.765803 8496 tablet_bootstrap.cc:492] T 14e325d8d3184c9192b8cebafb414190 P 47eabc135e9f4237b036097085e729d8: No bootstrap required, opened a new log
I20240501 14:08:53.765877 8496 ts_tablet_manager.cc:1367] T 14e325d8d3184c9192b8cebafb414190 P 47eabc135e9f4237b036097085e729d8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:08:53.765944 8494 tablet_bootstrap.cc:492] T 14e325d8d3184c9192b8cebafb414190 P ae52f888f5094416a24cb3dffbff03c4: No bootstrap required, opened a new log
I20240501 14:08:53.766016 8494 ts_tablet_manager.cc:1367] T 14e325d8d3184c9192b8cebafb414190 P ae52f888f5094416a24cb3dffbff03c4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:08:53.766139 8504 log.cc:824] T 2cf2f85819584fdf872f6c590d11f873 P 9e44efa72e7d4421b284ae74d1cb9634: Log is configured to *not* fsync() on all Append() calls
I20240501 14:08:53.766359 8496 raft_consensus.cc:348] T 14e325d8d3184c9192b8cebafb414190 P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } }
I20240501 14:08:53.766465 8496 raft_consensus.cc:374] T 14e325d8d3184c9192b8cebafb414190 P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:53.766502 8496 raft_consensus.cc:729] T 14e325d8d3184c9192b8cebafb414190 P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47eabc135e9f4237b036097085e729d8, State: Initialized, Role: FOLLOWER
I20240501 14:08:53.766482 8494 raft_consensus.cc:348] T 14e325d8d3184c9192b8cebafb414190 P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } }
I20240501 14:08:53.766578 8494 raft_consensus.cc:374] T 14e325d8d3184c9192b8cebafb414190 P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:53.766628 8494 raft_consensus.cc:729] T 14e325d8d3184c9192b8cebafb414190 P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae52f888f5094416a24cb3dffbff03c4, State: Initialized, Role: FOLLOWER
I20240501 14:08:53.766623 8496 consensus_queue.cc:260] T 14e325d8d3184c9192b8cebafb414190 P 47eabc135e9f4237b036097085e729d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } }
I20240501 14:08:53.766754 8494 consensus_queue.cc:260] T 14e325d8d3184c9192b8cebafb414190 P ae52f888f5094416a24cb3dffbff03c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } }
I20240501 14:08:53.766892 8496 ts_tablet_manager.cc:1398] T 14e325d8d3184c9192b8cebafb414190 P 47eabc135e9f4237b036097085e729d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:53.767022 8494 ts_tablet_manager.cc:1398] T 14e325d8d3184c9192b8cebafb414190 P ae52f888f5094416a24cb3dffbff03c4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:08:53.768571 8504 tablet_bootstrap.cc:492] T 2cf2f85819584fdf872f6c590d11f873 P 9e44efa72e7d4421b284ae74d1cb9634: No bootstrap required, opened a new log
I20240501 14:08:53.768733 8504 ts_tablet_manager.cc:1367] T 2cf2f85819584fdf872f6c590d11f873 P 9e44efa72e7d4421b284ae74d1cb9634: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240501 14:08:53.771515 8504 raft_consensus.cc:348] T 2cf2f85819584fdf872f6c590d11f873 P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:08:53.771684 8504 raft_consensus.cc:374] T 2cf2f85819584fdf872f6c590d11f873 P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:53.771741 8504 raft_consensus.cc:729] T 2cf2f85819584fdf872f6c590d11f873 P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e44efa72e7d4421b284ae74d1cb9634, State: Initialized, Role: FOLLOWER
I20240501 14:08:53.772130 8504 consensus_queue.cc:260] T 2cf2f85819584fdf872f6c590d11f873 P 9e44efa72e7d4421b284ae74d1cb9634 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:08:53.772884 8504 ts_tablet_manager.cc:1398] T 2cf2f85819584fdf872f6c590d11f873 P 9e44efa72e7d4421b284ae74d1cb9634: Time spent starting tablet: real 0.004s user 0.000s sys 0.005s
I20240501 14:08:53.773113 8504 tablet_bootstrap.cc:492] T 14e325d8d3184c9192b8cebafb414190 P 9e44efa72e7d4421b284ae74d1cb9634: Bootstrap starting.
I20240501 14:08:53.774076 8504 tablet_bootstrap.cc:654] T 14e325d8d3184c9192b8cebafb414190 P 9e44efa72e7d4421b284ae74d1cb9634: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:53.775255 8504 tablet_bootstrap.cc:492] T 14e325d8d3184c9192b8cebafb414190 P 9e44efa72e7d4421b284ae74d1cb9634: No bootstrap required, opened a new log
I20240501 14:08:53.775334 8504 ts_tablet_manager.cc:1367] T 14e325d8d3184c9192b8cebafb414190 P 9e44efa72e7d4421b284ae74d1cb9634: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:08:53.775882 8504 raft_consensus.cc:348] T 14e325d8d3184c9192b8cebafb414190 P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } }
I20240501 14:08:53.775983 8504 raft_consensus.cc:374] T 14e325d8d3184c9192b8cebafb414190 P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:53.776016 8504 raft_consensus.cc:729] T 14e325d8d3184c9192b8cebafb414190 P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e44efa72e7d4421b284ae74d1cb9634, State: Initialized, Role: FOLLOWER
I20240501 14:08:53.776152 8504 consensus_queue.cc:260] T 14e325d8d3184c9192b8cebafb414190 P 9e44efa72e7d4421b284ae74d1cb9634 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } }
I20240501 14:08:53.776424 8504 ts_tablet_manager.cc:1398] T 14e325d8d3184c9192b8cebafb414190 P 9e44efa72e7d4421b284ae74d1cb9634: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240501 14:08:53.799695 7558 tablet.cc:2369] T 14e325d8d3184c9192b8cebafb414190 P ae52f888f5094416a24cb3dffbff03c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:08:53.830366 8507 raft_consensus.cc:482] T 2cf2f85819584fdf872f6c590d11f873 P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:53.830595 8507 raft_consensus.cc:504] T 2cf2f85819584fdf872f6c590d11f873 P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:08:53.831725 8507 leader_election.cc:288] T 2cf2f85819584fdf872f6c590d11f873 P ae52f888f5094416a24cb3dffbff03c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ae5a87ca07b40a788a42734b24f66c6 (127.7.39.131:34697), 9e44efa72e7d4421b284ae74d1cb9634 (127.7.39.130:46433)
I20240501 14:08:53.835780 7773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cf2f85819584fdf872f6c590d11f873" candidate_uuid: "ae52f888f5094416a24cb3dffbff03c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ae5a87ca07b40a788a42734b24f66c6" is_pre_election: true
I20240501 14:08:53.835812 7642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cf2f85819584fdf872f6c590d11f873" candidate_uuid: "ae52f888f5094416a24cb3dffbff03c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e44efa72e7d4421b284ae74d1cb9634" is_pre_election: true
I20240501 14:08:53.836062 7773 raft_consensus.cc:2454] T 2cf2f85819584fdf872f6c590d11f873 P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae52f888f5094416a24cb3dffbff03c4 in term 0.
I20240501 14:08:53.836076 7642 raft_consensus.cc:2454] T 2cf2f85819584fdf872f6c590d11f873 P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae52f888f5094416a24cb3dffbff03c4 in term 0.
I20240501 14:08:53.836433 7446 leader_election.cc:302] T 2cf2f85819584fdf872f6c590d11f873 P ae52f888f5094416a24cb3dffbff03c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e44efa72e7d4421b284ae74d1cb9634, ae52f888f5094416a24cb3dffbff03c4; no voters:
I20240501 14:08:53.836642 8507 raft_consensus.cc:2781] T 2cf2f85819584fdf872f6c590d11f873 P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:53.836716 8507 raft_consensus.cc:482] T 2cf2f85819584fdf872f6c590d11f873 P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:53.836768 8507 raft_consensus.cc:3037] T 2cf2f85819584fdf872f6c590d11f873 P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:53.837651 8507 raft_consensus.cc:504] T 2cf2f85819584fdf872f6c590d11f873 P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:08:53.837958 8507 leader_election.cc:288] T 2cf2f85819584fdf872f6c590d11f873 P ae52f888f5094416a24cb3dffbff03c4 [CANDIDATE]: Term 1 election: Requested vote from peers 9ae5a87ca07b40a788a42734b24f66c6 (127.7.39.131:34697), 9e44efa72e7d4421b284ae74d1cb9634 (127.7.39.130:46433)
I20240501 14:08:53.838366 7773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cf2f85819584fdf872f6c590d11f873" candidate_uuid: "ae52f888f5094416a24cb3dffbff03c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ae5a87ca07b40a788a42734b24f66c6"
I20240501 14:08:53.838373 7642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cf2f85819584fdf872f6c590d11f873" candidate_uuid: "ae52f888f5094416a24cb3dffbff03c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e44efa72e7d4421b284ae74d1cb9634"
I20240501 14:08:53.838469 7642 raft_consensus.cc:3037] T 2cf2f85819584fdf872f6c590d11f873 P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:53.838479 7773 raft_consensus.cc:3037] T 2cf2f85819584fdf872f6c590d11f873 P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:53.839411 7773 raft_consensus.cc:2454] T 2cf2f85819584fdf872f6c590d11f873 P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae52f888f5094416a24cb3dffbff03c4 in term 1.
I20240501 14:08:53.839427 7642 raft_consensus.cc:2454] T 2cf2f85819584fdf872f6c590d11f873 P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae52f888f5094416a24cb3dffbff03c4 in term 1.
I20240501 14:08:53.839676 7446 leader_election.cc:302] T 2cf2f85819584fdf872f6c590d11f873 P ae52f888f5094416a24cb3dffbff03c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e44efa72e7d4421b284ae74d1cb9634, ae52f888f5094416a24cb3dffbff03c4; no voters:
I20240501 14:08:53.839857 8507 raft_consensus.cc:2781] T 2cf2f85819584fdf872f6c590d11f873 P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:53.839946 8507 raft_consensus.cc:686] T 2cf2f85819584fdf872f6c590d11f873 P ae52f888f5094416a24cb3dffbff03c4 [term 1 LEADER]: Becoming Leader. State: Replica: ae52f888f5094416a24cb3dffbff03c4, State: Running, Role: LEADER
I20240501 14:08:53.840258 8507 consensus_queue.cc:237] T 2cf2f85819584fdf872f6c590d11f873 P ae52f888f5094416a24cb3dffbff03c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:08:53.843132 7369 catalog_manager.cc:5484] T 2cf2f85819584fdf872f6c590d11f873 P ae52f888f5094416a24cb3dffbff03c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129). New cstate: current_term: 1 leader_uuid: "ae52f888f5094416a24cb3dffbff03c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } health_report { overall_health: UNKNOWN } } }
I20240501 14:08:53.904251 8505 raft_consensus.cc:482] T 14e325d8d3184c9192b8cebafb414190 P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:53.904476 8505 raft_consensus.cc:504] T 14e325d8d3184c9192b8cebafb414190 P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } }
I20240501 14:08:53.905658 8505 leader_election.cc:288] T 14e325d8d3184c9192b8cebafb414190 P 47eabc135e9f4237b036097085e729d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e44efa72e7d4421b284ae74d1cb9634 (127.7.39.130:46433), ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083)
I20240501 14:08:53.909996 7642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14e325d8d3184c9192b8cebafb414190" candidate_uuid: "47eabc135e9f4237b036097085e729d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e44efa72e7d4421b284ae74d1cb9634" is_pre_election: true
I20240501 14:08:53.910079 7511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14e325d8d3184c9192b8cebafb414190" candidate_uuid: "47eabc135e9f4237b036097085e729d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae52f888f5094416a24cb3dffbff03c4" is_pre_election: true
I20240501 14:08:53.910187 7642 raft_consensus.cc:2454] T 14e325d8d3184c9192b8cebafb414190 P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47eabc135e9f4237b036097085e729d8 in term 0.
I20240501 14:08:53.910315 7511 raft_consensus.cc:2454] T 14e325d8d3184c9192b8cebafb414190 P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47eabc135e9f4237b036097085e729d8 in term 0.
I20240501 14:08:53.910627 7839 leader_election.cc:302] T 14e325d8d3184c9192b8cebafb414190 P 47eabc135e9f4237b036097085e729d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47eabc135e9f4237b036097085e729d8, 9e44efa72e7d4421b284ae74d1cb9634; no voters:
I20240501 14:08:53.910902 8505 raft_consensus.cc:2781] T 14e325d8d3184c9192b8cebafb414190 P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:53.910987 8505 raft_consensus.cc:482] T 14e325d8d3184c9192b8cebafb414190 P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:53.911023 8505 raft_consensus.cc:3037] T 14e325d8d3184c9192b8cebafb414190 P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:53.912029 8505 raft_consensus.cc:504] T 14e325d8d3184c9192b8cebafb414190 P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } }
I20240501 14:08:53.912402 8505 leader_election.cc:288] T 14e325d8d3184c9192b8cebafb414190 P 47eabc135e9f4237b036097085e729d8 [CANDIDATE]: Term 1 election: Requested vote from peers 9e44efa72e7d4421b284ae74d1cb9634 (127.7.39.130:46433), ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083)
I20240501 14:08:53.912758 7642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14e325d8d3184c9192b8cebafb414190" candidate_uuid: "47eabc135e9f4237b036097085e729d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e44efa72e7d4421b284ae74d1cb9634"
I20240501 14:08:53.912817 7511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14e325d8d3184c9192b8cebafb414190" candidate_uuid: "47eabc135e9f4237b036097085e729d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae52f888f5094416a24cb3dffbff03c4"
I20240501 14:08:53.912930 7511 raft_consensus.cc:3037] T 14e325d8d3184c9192b8cebafb414190 P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:53.912933 7642 raft_consensus.cc:3037] T 14e325d8d3184c9192b8cebafb414190 P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:53.913955 7642 raft_consensus.cc:2454] T 14e325d8d3184c9192b8cebafb414190 P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47eabc135e9f4237b036097085e729d8 in term 1.
I20240501 14:08:53.913955 7511 raft_consensus.cc:2454] T 14e325d8d3184c9192b8cebafb414190 P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47eabc135e9f4237b036097085e729d8 in term 1.
I20240501 14:08:53.914245 7839 leader_election.cc:302] T 14e325d8d3184c9192b8cebafb414190 P 47eabc135e9f4237b036097085e729d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47eabc135e9f4237b036097085e729d8, 9e44efa72e7d4421b284ae74d1cb9634; no voters:
I20240501 14:08:53.914407 8505 raft_consensus.cc:2781] T 14e325d8d3184c9192b8cebafb414190 P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:53.914647 8505 raft_consensus.cc:686] T 14e325d8d3184c9192b8cebafb414190 P 47eabc135e9f4237b036097085e729d8 [term 1 LEADER]: Becoming Leader. State: Replica: 47eabc135e9f4237b036097085e729d8, State: Running, Role: LEADER
I20240501 14:08:53.914961 8505 consensus_queue.cc:237] T 14e325d8d3184c9192b8cebafb414190 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } }
I20240501 14:08:53.918061 7369 catalog_manager.cc:5484] T 14e325d8d3184c9192b8cebafb414190 P 47eabc135e9f4237b036097085e729d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 47eabc135e9f4237b036097085e729d8 (127.7.39.132). New cstate: current_term: 1 leader_uuid: "47eabc135e9f4237b036097085e729d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } health_report { overall_health: UNKNOWN } } }
W20240501 14:08:53.924654 7820 tablet.cc:2369] T 2cf2f85819584fdf872f6c590d11f873 P 9ae5a87ca07b40a788a42734b24f66c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:08:53.941270 7951 tablet.cc:2369] T 14e325d8d3184c9192b8cebafb414190 P 47eabc135e9f4237b036097085e729d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:08:53.991429 7689 tablet.cc:2369] T 14e325d8d3184c9192b8cebafb414190 P 9e44efa72e7d4421b284ae74d1cb9634: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:08:54.120088 8507 raft_consensus.cc:482] T ceee52d679bf4d8e8896a6c0dde0519b P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:54.120249 8507 raft_consensus.cc:504] T ceee52d679bf4d8e8896a6c0dde0519b P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:08:54.120795 8507 leader_election.cc:288] T ceee52d679bf4d8e8896a6c0dde0519b P ae52f888f5094416a24cb3dffbff03c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47eabc135e9f4237b036097085e729d8 (127.7.39.132:42369), 9ae5a87ca07b40a788a42734b24f66c6 (127.7.39.131:34697)
I20240501 14:08:54.121376 7773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceee52d679bf4d8e8896a6c0dde0519b" candidate_uuid: "ae52f888f5094416a24cb3dffbff03c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ae5a87ca07b40a788a42734b24f66c6" is_pre_election: true
I20240501 14:08:54.121527 7773 raft_consensus.cc:2454] T ceee52d679bf4d8e8896a6c0dde0519b P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae52f888f5094416a24cb3dffbff03c4 in term 0.
I20240501 14:08:54.121850 7446 leader_election.cc:302] T ceee52d679bf4d8e8896a6c0dde0519b P ae52f888f5094416a24cb3dffbff03c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ae5a87ca07b40a788a42734b24f66c6, ae52f888f5094416a24cb3dffbff03c4; no voters:
I20240501 14:08:54.122020 8507 raft_consensus.cc:2781] T ceee52d679bf4d8e8896a6c0dde0519b P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:54.122084 8507 raft_consensus.cc:482] T ceee52d679bf4d8e8896a6c0dde0519b P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:54.122133 8507 raft_consensus.cc:3037] T ceee52d679bf4d8e8896a6c0dde0519b P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:54.125068 7904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceee52d679bf4d8e8896a6c0dde0519b" candidate_uuid: "ae52f888f5094416a24cb3dffbff03c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47eabc135e9f4237b036097085e729d8" is_pre_election: true
I20240501 14:08:54.125284 7904 raft_consensus.cc:2454] T ceee52d679bf4d8e8896a6c0dde0519b P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae52f888f5094416a24cb3dffbff03c4 in term 0.
I20240501 14:08:54.165964 8507 raft_consensus.cc:504] T ceee52d679bf4d8e8896a6c0dde0519b P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:08:54.166532 8507 leader_election.cc:288] T ceee52d679bf4d8e8896a6c0dde0519b P ae52f888f5094416a24cb3dffbff03c4 [CANDIDATE]: Term 1 election: Requested vote from peers 47eabc135e9f4237b036097085e729d8 (127.7.39.132:42369), 9ae5a87ca07b40a788a42734b24f66c6 (127.7.39.131:34697)
I20240501 14:08:54.167112 7773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceee52d679bf4d8e8896a6c0dde0519b" candidate_uuid: "ae52f888f5094416a24cb3dffbff03c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ae5a87ca07b40a788a42734b24f66c6"
I20240501 14:08:54.167135 7904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceee52d679bf4d8e8896a6c0dde0519b" candidate_uuid: "ae52f888f5094416a24cb3dffbff03c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47eabc135e9f4237b036097085e729d8"
I20240501 14:08:54.167263 7773 raft_consensus.cc:3037] T ceee52d679bf4d8e8896a6c0dde0519b P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:54.167299 7904 raft_consensus.cc:3037] T ceee52d679bf4d8e8896a6c0dde0519b P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:54.168109 7773 raft_consensus.cc:2454] T ceee52d679bf4d8e8896a6c0dde0519b P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae52f888f5094416a24cb3dffbff03c4 in term 1.
I20240501 14:08:54.168473 7446 leader_election.cc:302] T ceee52d679bf4d8e8896a6c0dde0519b P ae52f888f5094416a24cb3dffbff03c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ae5a87ca07b40a788a42734b24f66c6, ae52f888f5094416a24cb3dffbff03c4; no voters:
I20240501 14:08:54.168674 8507 raft_consensus.cc:2781] T ceee52d679bf4d8e8896a6c0dde0519b P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:54.168771 8507 raft_consensus.cc:686] T ceee52d679bf4d8e8896a6c0dde0519b P ae52f888f5094416a24cb3dffbff03c4 [term 1 LEADER]: Becoming Leader. State: Replica: ae52f888f5094416a24cb3dffbff03c4, State: Running, Role: LEADER
I20240501 14:08:54.168942 8507 consensus_queue.cc:237] T ceee52d679bf4d8e8896a6c0dde0519b P ae52f888f5094416a24cb3dffbff03c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:08:54.169143 7904 raft_consensus.cc:2454] T ceee52d679bf4d8e8896a6c0dde0519b P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae52f888f5094416a24cb3dffbff03c4 in term 1.
I20240501 14:08:54.170773 7369 catalog_manager.cc:5484] T ceee52d679bf4d8e8896a6c0dde0519b P ae52f888f5094416a24cb3dffbff03c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129). New cstate: current_term: 1 leader_uuid: "ae52f888f5094416a24cb3dffbff03c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } health_report { overall_health: UNKNOWN } } }
W20240501 14:08:54.297720 8521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:54.298043 8521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:54.301570 8521 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:08:54.305071 8507 consensus_queue.cc:1035] T 2cf2f85819584fdf872f6c590d11f873 P ae52f888f5094416a24cb3dffbff03c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:54.310719 8501 consensus_queue.cc:1035] T 2cf2f85819584fdf872f6c590d11f873 P ae52f888f5094416a24cb3dffbff03c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
I20240501 14:08:54.407689 7772 raft_consensus.cc:1261] T ceee52d679bf4d8e8896a6c0dde0519b P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Refusing update from remote peer ae52f888f5094416a24cb3dffbff03c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:54.408288 8507 consensus_queue.cc:1035] T ceee52d679bf4d8e8896a6c0dde0519b P ae52f888f5094416a24cb3dffbff03c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:54.413373 7904 raft_consensus.cc:1261] T ceee52d679bf4d8e8896a6c0dde0519b P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Refusing update from remote peer ae52f888f5094416a24cb3dffbff03c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:54.413940 8501 consensus_queue.cc:1035] T ceee52d679bf4d8e8896a6c0dde0519b P ae52f888f5094416a24cb3dffbff03c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:54.421470 7511 raft_consensus.cc:1261] T 14e325d8d3184c9192b8cebafb414190 P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:54.423517 7642 raft_consensus.cc:1261] T 14e325d8d3184c9192b8cebafb414190 P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 FOLLOWER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:54.424015 8505 consensus_queue.cc:1035] T 14e325d8d3184c9192b8cebafb414190 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:54.424573 8505 consensus_queue.cc:1035] T 14e325d8d3184c9192b8cebafb414190 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:54.431031 8558 mvcc.cc:205] Tried to move back new op lower bound from 7022889100927750144 to 7022889099958308864. Current Snapshot: MvccSnapshot[applied={T|T < 7022889100927750144}]
I20240501 14:08:54.434836 8529 mvcc.cc:205] Tried to move back new op lower bound from 7022889100927750144 to 7022889099958308864. Current Snapshot: MvccSnapshot[applied={T|T < 7022889100927750144}]
I20240501 14:08:54.440711 8565 mvcc.cc:205] Tried to move back new op lower bound from 7022889100978909184 to 7022889098918928384. Current Snapshot: MvccSnapshot[applied={T|T < 7022889100978909184}]
I20240501 14:08:54.448314 8537 mvcc.cc:205] Tried to move back new op lower bound from 7022889100978909184 to 7022889098918928384. Current Snapshot: MvccSnapshot[applied={T|T < 7022889100978909184}]
INSERT report
rows total: 2000
time total: 130.218 ms
time per row: 0.0651088 ms
I20240501 14:08:54.484098 7369 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32870:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:08:54.489703 7753 tablet_service.cc:1467] Processing CreateTablet for tablet 0bfcc65d57eb4c3597a761f878a09da2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=95829980e415451998b6e8797247120f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:54.489725 7622 tablet_service.cc:1467] Processing CreateTablet for tablet 0bfcc65d57eb4c3597a761f878a09da2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=95829980e415451998b6e8797247120f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:54.490075 7753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bfcc65d57eb4c3597a761f878a09da2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:54.490082 7622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bfcc65d57eb4c3597a761f878a09da2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:54.490489 7883 tablet_service.cc:1467] Processing CreateTablet for tablet 4573be69270c489289af5fc65f67112d (DEFAULT_TABLE table=rebalance_test_table_1 [id=95829980e415451998b6e8797247120f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:54.490741 7883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4573be69270c489289af5fc65f67112d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:54.491120 7621 tablet_service.cc:1467] Processing CreateTablet for tablet 4573be69270c489289af5fc65f67112d (DEFAULT_TABLE table=rebalance_test_table_1 [id=95829980e415451998b6e8797247120f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:54.491367 7621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4573be69270c489289af5fc65f67112d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:54.491541 7489 tablet_service.cc:1467] Processing CreateTablet for tablet ae06bed9bd844a05ae2ac69148bf5bad (DEFAULT_TABLE table=rebalance_test_table_1 [id=95829980e415451998b6e8797247120f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:54.491791 7489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae06bed9bd844a05ae2ac69148bf5bad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:54.492749 7620 tablet_service.cc:1467] Processing CreateTablet for tablet ae06bed9bd844a05ae2ac69148bf5bad (DEFAULT_TABLE table=rebalance_test_table_1 [id=95829980e415451998b6e8797247120f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:54.492990 7620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae06bed9bd844a05ae2ac69148bf5bad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:54.493168 7884 tablet_service.cc:1467] Processing CreateTablet for tablet 0bfcc65d57eb4c3597a761f878a09da2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=95829980e415451998b6e8797247120f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:54.493402 7884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bfcc65d57eb4c3597a761f878a09da2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:54.494649 8585 tablet_bootstrap.cc:492] T ae06bed9bd844a05ae2ac69148bf5bad P ae52f888f5094416a24cb3dffbff03c4: Bootstrap starting.
I20240501 14:08:54.495594 8586 tablet_bootstrap.cc:492] T 0bfcc65d57eb4c3597a761f878a09da2 P 47eabc135e9f4237b036097085e729d8: Bootstrap starting.
I20240501 14:08:54.495908 8585 tablet_bootstrap.cc:654] T ae06bed9bd844a05ae2ac69148bf5bad P ae52f888f5094416a24cb3dffbff03c4: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:54.496275 7752 tablet_service.cc:1467] Processing CreateTablet for tablet ae06bed9bd844a05ae2ac69148bf5bad (DEFAULT_TABLE table=rebalance_test_table_1 [id=95829980e415451998b6e8797247120f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:54.496537 7752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae06bed9bd844a05ae2ac69148bf5bad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:54.496670 8586 tablet_bootstrap.cc:654] T 0bfcc65d57eb4c3597a761f878a09da2 P 47eabc135e9f4237b036097085e729d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:54.496702 7753 tablet_service.cc:1467] Processing CreateTablet for tablet 4573be69270c489289af5fc65f67112d (DEFAULT_TABLE table=rebalance_test_table_1 [id=95829980e415451998b6e8797247120f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:54.496908 7753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4573be69270c489289af5fc65f67112d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:54.497833 8586 tablet_bootstrap.cc:492] T 0bfcc65d57eb4c3597a761f878a09da2 P 47eabc135e9f4237b036097085e729d8: No bootstrap required, opened a new log
I20240501 14:08:54.497928 8586 ts_tablet_manager.cc:1367] T 0bfcc65d57eb4c3597a761f878a09da2 P 47eabc135e9f4237b036097085e729d8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:54.497968 8585 tablet_bootstrap.cc:492] T ae06bed9bd844a05ae2ac69148bf5bad P ae52f888f5094416a24cb3dffbff03c4: No bootstrap required, opened a new log
I20240501 14:08:54.498059 8585 ts_tablet_manager.cc:1367] T ae06bed9bd844a05ae2ac69148bf5bad P ae52f888f5094416a24cb3dffbff03c4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:08:54.498435 8586 raft_consensus.cc:348] T 0bfcc65d57eb4c3597a761f878a09da2 P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } }
I20240501 14:08:54.498538 8586 raft_consensus.cc:374] T 0bfcc65d57eb4c3597a761f878a09da2 P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:54.498570 8586 raft_consensus.cc:729] T 0bfcc65d57eb4c3597a761f878a09da2 P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47eabc135e9f4237b036097085e729d8, State: Initialized, Role: FOLLOWER
I20240501 14:08:54.498697 8586 consensus_queue.cc:260] T 0bfcc65d57eb4c3597a761f878a09da2 P 47eabc135e9f4237b036097085e729d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } }
I20240501 14:08:54.498798 8585 raft_consensus.cc:348] T ae06bed9bd844a05ae2ac69148bf5bad P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:08:54.498893 8585 raft_consensus.cc:374] T ae06bed9bd844a05ae2ac69148bf5bad P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:54.498924 8585 raft_consensus.cc:729] T ae06bed9bd844a05ae2ac69148bf5bad P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae52f888f5094416a24cb3dffbff03c4, State: Initialized, Role: FOLLOWER
I20240501 14:08:54.498997 8586 ts_tablet_manager.cc:1398] T 0bfcc65d57eb4c3597a761f878a09da2 P 47eabc135e9f4237b036097085e729d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:54.499074 8585 consensus_queue.cc:260] T ae06bed9bd844a05ae2ac69148bf5bad P ae52f888f5094416a24cb3dffbff03c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:08:54.499202 8586 tablet_bootstrap.cc:492] T 4573be69270c489289af5fc65f67112d P 47eabc135e9f4237b036097085e729d8: Bootstrap starting.
I20240501 14:08:54.499341 8585 ts_tablet_manager.cc:1398] T ae06bed9bd844a05ae2ac69148bf5bad P ae52f888f5094416a24cb3dffbff03c4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:54.499788 8584 tablet_bootstrap.cc:492] T 0bfcc65d57eb4c3597a761f878a09da2 P 9e44efa72e7d4421b284ae74d1cb9634: Bootstrap starting.
I20240501 14:08:54.500149 8586 tablet_bootstrap.cc:654] T 4573be69270c489289af5fc65f67112d P 47eabc135e9f4237b036097085e729d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:54.500905 8583 tablet_bootstrap.cc:492] T 0bfcc65d57eb4c3597a761f878a09da2 P 9ae5a87ca07b40a788a42734b24f66c6: Bootstrap starting.
I20240501 14:08:54.500993 8584 tablet_bootstrap.cc:654] T 0bfcc65d57eb4c3597a761f878a09da2 P 9e44efa72e7d4421b284ae74d1cb9634: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:54.502183 8583 tablet_bootstrap.cc:654] T 0bfcc65d57eb4c3597a761f878a09da2 P 9ae5a87ca07b40a788a42734b24f66c6: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:54.502882 8584 tablet_bootstrap.cc:492] T 0bfcc65d57eb4c3597a761f878a09da2 P 9e44efa72e7d4421b284ae74d1cb9634: No bootstrap required, opened a new log
I20240501 14:08:54.502976 8584 ts_tablet_manager.cc:1367] T 0bfcc65d57eb4c3597a761f878a09da2 P 9e44efa72e7d4421b284ae74d1cb9634: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:08:54.503409 8586 tablet_bootstrap.cc:492] T 4573be69270c489289af5fc65f67112d P 47eabc135e9f4237b036097085e729d8: No bootstrap required, opened a new log
I20240501 14:08:54.503484 8586 ts_tablet_manager.cc:1367] T 4573be69270c489289af5fc65f67112d P 47eabc135e9f4237b036097085e729d8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:08:54.503487 8583 tablet_bootstrap.cc:492] T 0bfcc65d57eb4c3597a761f878a09da2 P 9ae5a87ca07b40a788a42734b24f66c6: No bootstrap required, opened a new log
I20240501 14:08:54.503568 8583 ts_tablet_manager.cc:1367] T 0bfcc65d57eb4c3597a761f878a09da2 P 9ae5a87ca07b40a788a42734b24f66c6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:08:54.503541 8584 raft_consensus.cc:348] T 0bfcc65d57eb4c3597a761f878a09da2 P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } }
I20240501 14:08:54.503661 8584 raft_consensus.cc:374] T 0bfcc65d57eb4c3597a761f878a09da2 P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:54.503698 8584 raft_consensus.cc:729] T 0bfcc65d57eb4c3597a761f878a09da2 P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e44efa72e7d4421b284ae74d1cb9634, State: Initialized, Role: FOLLOWER
I20240501 14:08:54.503818 8584 consensus_queue.cc:260] T 0bfcc65d57eb4c3597a761f878a09da2 P 9e44efa72e7d4421b284ae74d1cb9634 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } }
I20240501 14:08:54.504086 8584 ts_tablet_manager.cc:1398] T 0bfcc65d57eb4c3597a761f878a09da2 P 9e44efa72e7d4421b284ae74d1cb9634: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:08:54.504047 8586 raft_consensus.cc:348] T 4573be69270c489289af5fc65f67112d P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:08:54.504139 8586 raft_consensus.cc:374] T 4573be69270c489289af5fc65f67112d P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:54.504164 8586 raft_consensus.cc:729] T 4573be69270c489289af5fc65f67112d P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47eabc135e9f4237b036097085e729d8, State: Initialized, Role: FOLLOWER
I20240501 14:08:54.504238 8584 tablet_bootstrap.cc:492] T 4573be69270c489289af5fc65f67112d P 9e44efa72e7d4421b284ae74d1cb9634: Bootstrap starting.
I20240501 14:08:54.504271 8586 consensus_queue.cc:260] T 4573be69270c489289af5fc65f67112d P 47eabc135e9f4237b036097085e729d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:08:54.504382 8583 raft_consensus.cc:348] T 0bfcc65d57eb4c3597a761f878a09da2 P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } }
I20240501 14:08:54.504483 8583 raft_consensus.cc:374] T 0bfcc65d57eb4c3597a761f878a09da2 P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:54.504515 8583 raft_consensus.cc:729] T 0bfcc65d57eb4c3597a761f878a09da2 P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ae5a87ca07b40a788a42734b24f66c6, State: Initialized, Role: FOLLOWER
I20240501 14:08:54.504621 8586 ts_tablet_manager.cc:1398] T 4573be69270c489289af5fc65f67112d P 47eabc135e9f4237b036097085e729d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:54.504668 8583 consensus_queue.cc:260] T 0bfcc65d57eb4c3597a761f878a09da2 P 9ae5a87ca07b40a788a42734b24f66c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } }
I20240501 14:08:54.505132 8584 tablet_bootstrap.cc:654] T 4573be69270c489289af5fc65f67112d P 9e44efa72e7d4421b284ae74d1cb9634: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:54.506129 8583 ts_tablet_manager.cc:1398] T 0bfcc65d57eb4c3597a761f878a09da2 P 9ae5a87ca07b40a788a42734b24f66c6: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:08:54.506374 8583 tablet_bootstrap.cc:492] T ae06bed9bd844a05ae2ac69148bf5bad P 9ae5a87ca07b40a788a42734b24f66c6: Bootstrap starting.
I20240501 14:08:54.507282 8583 tablet_bootstrap.cc:654] T ae06bed9bd844a05ae2ac69148bf5bad P 9ae5a87ca07b40a788a42734b24f66c6: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:54.507344 8584 tablet_bootstrap.cc:492] T 4573be69270c489289af5fc65f67112d P 9e44efa72e7d4421b284ae74d1cb9634: No bootstrap required, opened a new log
I20240501 14:08:54.507473 8584 ts_tablet_manager.cc:1367] T 4573be69270c489289af5fc65f67112d P 9e44efa72e7d4421b284ae74d1cb9634: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:08:54.507948 8584 raft_consensus.cc:348] T 4573be69270c489289af5fc65f67112d P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:08:54.508069 8584 raft_consensus.cc:374] T 4573be69270c489289af5fc65f67112d P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:54.508128 8584 raft_consensus.cc:729] T 4573be69270c489289af5fc65f67112d P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e44efa72e7d4421b284ae74d1cb9634, State: Initialized, Role: FOLLOWER
I20240501 14:08:54.508267 8584 consensus_queue.cc:260] T 4573be69270c489289af5fc65f67112d P 9e44efa72e7d4421b284ae74d1cb9634 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:08:54.508570 8584 ts_tablet_manager.cc:1398] T 4573be69270c489289af5fc65f67112d P 9e44efa72e7d4421b284ae74d1cb9634: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:54.508678 8583 tablet_bootstrap.cc:492] T ae06bed9bd844a05ae2ac69148bf5bad P 9ae5a87ca07b40a788a42734b24f66c6: No bootstrap required, opened a new log
I20240501 14:08:54.508731 8584 tablet_bootstrap.cc:492] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634: Bootstrap starting.
I20240501 14:08:54.508756 8583 ts_tablet_manager.cc:1367] T ae06bed9bd844a05ae2ac69148bf5bad P 9ae5a87ca07b40a788a42734b24f66c6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:54.509198 8583 raft_consensus.cc:348] T ae06bed9bd844a05ae2ac69148bf5bad P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:08:54.509301 8583 raft_consensus.cc:374] T ae06bed9bd844a05ae2ac69148bf5bad P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:54.509331 8583 raft_consensus.cc:729] T ae06bed9bd844a05ae2ac69148bf5bad P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ae5a87ca07b40a788a42734b24f66c6, State: Initialized, Role: FOLLOWER
I20240501 14:08:54.509454 8583 consensus_queue.cc:260] T ae06bed9bd844a05ae2ac69148bf5bad P 9ae5a87ca07b40a788a42734b24f66c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:08:54.509617 8584 tablet_bootstrap.cc:654] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:54.509778 8583 ts_tablet_manager.cc:1398] T ae06bed9bd844a05ae2ac69148bf5bad P 9ae5a87ca07b40a788a42734b24f66c6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:54.509995 8583 tablet_bootstrap.cc:492] T 4573be69270c489289af5fc65f67112d P 9ae5a87ca07b40a788a42734b24f66c6: Bootstrap starting.
I20240501 14:08:54.510900 8583 tablet_bootstrap.cc:654] T 4573be69270c489289af5fc65f67112d P 9ae5a87ca07b40a788a42734b24f66c6: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:54.511174 8584 tablet_bootstrap.cc:492] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634: No bootstrap required, opened a new log
I20240501 14:08:54.511250 8584 ts_tablet_manager.cc:1367] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:08:54.511979 8584 raft_consensus.cc:348] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:08:54.512095 8584 raft_consensus.cc:374] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:54.512135 8584 raft_consensus.cc:729] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e44efa72e7d4421b284ae74d1cb9634, State: Initialized, Role: FOLLOWER
I20240501 14:08:54.512142 8583 tablet_bootstrap.cc:492] T 4573be69270c489289af5fc65f67112d P 9ae5a87ca07b40a788a42734b24f66c6: No bootstrap required, opened a new log
I20240501 14:08:54.512223 8583 ts_tablet_manager.cc:1367] T 4573be69270c489289af5fc65f67112d P 9ae5a87ca07b40a788a42734b24f66c6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:54.512262 8584 consensus_queue.cc:260] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:08:54.512575 8584 ts_tablet_manager.cc:1398] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:54.512740 8583 raft_consensus.cc:348] T 4573be69270c489289af5fc65f67112d P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:08:54.512812 8583 raft_consensus.cc:374] T 4573be69270c489289af5fc65f67112d P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:54.512831 8583 raft_consensus.cc:729] T 4573be69270c489289af5fc65f67112d P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ae5a87ca07b40a788a42734b24f66c6, State: Initialized, Role: FOLLOWER
I20240501 14:08:54.512929 8583 consensus_queue.cc:260] T 4573be69270c489289af5fc65f67112d P 9ae5a87ca07b40a788a42734b24f66c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:08:54.513259 8583 ts_tablet_manager.cc:1398] T 4573be69270c489289af5fc65f67112d P 9ae5a87ca07b40a788a42734b24f66c6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:54.514035 8535 raft_consensus.cc:482] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:54.514122 8535 raft_consensus.cc:504] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:08:54.515336 8535 leader_election.cc:288] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083), 9ae5a87ca07b40a788a42734b24f66c6 (127.7.39.131:34697)
I20240501 14:08:54.519444 7772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae06bed9bd844a05ae2ac69148bf5bad" candidate_uuid: "9e44efa72e7d4421b284ae74d1cb9634" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ae5a87ca07b40a788a42734b24f66c6" is_pre_election: true
I20240501 14:08:54.519536 7511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae06bed9bd844a05ae2ac69148bf5bad" candidate_uuid: "9e44efa72e7d4421b284ae74d1cb9634" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae52f888f5094416a24cb3dffbff03c4" is_pre_election: true
I20240501 14:08:54.519609 7772 raft_consensus.cc:2454] T ae06bed9bd844a05ae2ac69148bf5bad P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e44efa72e7d4421b284ae74d1cb9634 in term 0.
I20240501 14:08:54.519670 7511 raft_consensus.cc:2454] T ae06bed9bd844a05ae2ac69148bf5bad P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e44efa72e7d4421b284ae74d1cb9634 in term 0.
I20240501 14:08:54.519979 7577 leader_election.cc:302] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ae5a87ca07b40a788a42734b24f66c6, 9e44efa72e7d4421b284ae74d1cb9634; no voters:
I20240501 14:08:54.520220 8535 raft_consensus.cc:2781] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:54.520287 8535 raft_consensus.cc:482] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:54.520325 8535 raft_consensus.cc:3037] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:54.561594 8535 raft_consensus.cc:504] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:08:54.562094 8535 leader_election.cc:288] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 [CANDIDATE]: Term 1 election: Requested vote from peers ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083), 9ae5a87ca07b40a788a42734b24f66c6 (127.7.39.131:34697)
I20240501 14:08:54.562500 7511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae06bed9bd844a05ae2ac69148bf5bad" candidate_uuid: "9e44efa72e7d4421b284ae74d1cb9634" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae52f888f5094416a24cb3dffbff03c4"
I20240501 14:08:54.562613 7772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae06bed9bd844a05ae2ac69148bf5bad" candidate_uuid: "9e44efa72e7d4421b284ae74d1cb9634" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ae5a87ca07b40a788a42734b24f66c6"
I20240501 14:08:54.562659 7511 raft_consensus.cc:3037] T ae06bed9bd844a05ae2ac69148bf5bad P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:54.562738 7772 raft_consensus.cc:3037] T ae06bed9bd844a05ae2ac69148bf5bad P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:54.563900 7511 raft_consensus.cc:2454] T ae06bed9bd844a05ae2ac69148bf5bad P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e44efa72e7d4421b284ae74d1cb9634 in term 1.
I20240501 14:08:54.564265 7772 raft_consensus.cc:2454] T ae06bed9bd844a05ae2ac69148bf5bad P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e44efa72e7d4421b284ae74d1cb9634 in term 1.
I20240501 14:08:54.564340 7577 leader_election.cc:302] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e44efa72e7d4421b284ae74d1cb9634, ae52f888f5094416a24cb3dffbff03c4; no voters:
I20240501 14:08:54.564532 8535 raft_consensus.cc:2781] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:54.564793 8535 raft_consensus.cc:686] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 LEADER]: Becoming Leader. State: Replica: 9e44efa72e7d4421b284ae74d1cb9634, State: Running, Role: LEADER
I20240501 14:08:54.565143 8535 consensus_queue.cc:237] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:08:54.567457 7369 catalog_manager.cc:5484] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e44efa72e7d4421b284ae74d1cb9634 (127.7.39.130). New cstate: current_term: 1 leader_uuid: "9e44efa72e7d4421b284ae74d1cb9634" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } health_report { overall_health: UNKNOWN } } }
I20240501 14:08:54.567929 8535 raft_consensus.cc:482] T 4573be69270c489289af5fc65f67112d P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:54.568022 8535 raft_consensus.cc:504] T 4573be69270c489289af5fc65f67112d P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:08:54.568524 8535 leader_election.cc:288] T 4573be69270c489289af5fc65f67112d P 9e44efa72e7d4421b284ae74d1cb9634 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47eabc135e9f4237b036097085e729d8 (127.7.39.132:42369), 9ae5a87ca07b40a788a42734b24f66c6 (127.7.39.131:34697)
I20240501 14:08:54.568967 7772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4573be69270c489289af5fc65f67112d" candidate_uuid: "9e44efa72e7d4421b284ae74d1cb9634" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ae5a87ca07b40a788a42734b24f66c6" is_pre_election: true
I20240501 14:08:54.569108 7772 raft_consensus.cc:2454] T 4573be69270c489289af5fc65f67112d P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e44efa72e7d4421b284ae74d1cb9634 in term 0.
I20240501 14:08:54.569447 7577 leader_election.cc:302] T 4573be69270c489289af5fc65f67112d P 9e44efa72e7d4421b284ae74d1cb9634 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ae5a87ca07b40a788a42734b24f66c6, 9e44efa72e7d4421b284ae74d1cb9634; no voters:
I20240501 14:08:54.569664 8535 raft_consensus.cc:2781] T 4573be69270c489289af5fc65f67112d P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:54.569765 8535 raft_consensus.cc:482] T 4573be69270c489289af5fc65f67112d P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:54.569878 8535 raft_consensus.cc:3037] T 4573be69270c489289af5fc65f67112d P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:54.572077 7904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4573be69270c489289af5fc65f67112d" candidate_uuid: "9e44efa72e7d4421b284ae74d1cb9634" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47eabc135e9f4237b036097085e729d8" is_pre_election: true
I20240501 14:08:54.572222 7904 raft_consensus.cc:2454] T 4573be69270c489289af5fc65f67112d P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e44efa72e7d4421b284ae74d1cb9634 in term 0.
I20240501 14:08:54.578765 8567 raft_consensus.cc:482] T 0bfcc65d57eb4c3597a761f878a09da2 P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:54.578902 8567 raft_consensus.cc:504] T 0bfcc65d57eb4c3597a761f878a09da2 P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } }
I20240501 14:08:54.579404 8567 leader_election.cc:288] T 0bfcc65d57eb4c3597a761f878a09da2 P 47eabc135e9f4237b036097085e729d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ae5a87ca07b40a788a42734b24f66c6 (127.7.39.131:34697), 9e44efa72e7d4421b284ae74d1cb9634 (127.7.39.130:46433)
I20240501 14:08:54.580272 7642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bfcc65d57eb4c3597a761f878a09da2" candidate_uuid: "47eabc135e9f4237b036097085e729d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e44efa72e7d4421b284ae74d1cb9634" is_pre_election: true
I20240501 14:08:54.580433 7642 raft_consensus.cc:2454] T 0bfcc65d57eb4c3597a761f878a09da2 P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47eabc135e9f4237b036097085e729d8 in term 0.
I20240501 14:08:54.580739 7839 leader_election.cc:302] T 0bfcc65d57eb4c3597a761f878a09da2 P 47eabc135e9f4237b036097085e729d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47eabc135e9f4237b036097085e729d8, 9e44efa72e7d4421b284ae74d1cb9634; no voters:
I20240501 14:08:54.580927 8567 raft_consensus.cc:2781] T 0bfcc65d57eb4c3597a761f878a09da2 P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:54.580987 8567 raft_consensus.cc:482] T 0bfcc65d57eb4c3597a761f878a09da2 P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:54.581025 8567 raft_consensus.cc:3037] T 0bfcc65d57eb4c3597a761f878a09da2 P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:54.583238 7772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bfcc65d57eb4c3597a761f878a09da2" candidate_uuid: "47eabc135e9f4237b036097085e729d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ae5a87ca07b40a788a42734b24f66c6" is_pre_election: true
I20240501 14:08:54.583370 7772 raft_consensus.cc:2454] T 0bfcc65d57eb4c3597a761f878a09da2 P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47eabc135e9f4237b036097085e729d8 in term 0.
I20240501 14:08:54.663161 8535 raft_consensus.cc:504] T 4573be69270c489289af5fc65f67112d P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:08:54.663161 8567 raft_consensus.cc:504] T 0bfcc65d57eb4c3597a761f878a09da2 P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } }
I20240501 14:08:54.663733 8567 leader_election.cc:288] T 0bfcc65d57eb4c3597a761f878a09da2 P 47eabc135e9f4237b036097085e729d8 [CANDIDATE]: Term 1 election: Requested vote from peers 9ae5a87ca07b40a788a42734b24f66c6 (127.7.39.131:34697), 9e44efa72e7d4421b284ae74d1cb9634 (127.7.39.130:46433)
I20240501 14:08:54.664240 7904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4573be69270c489289af5fc65f67112d" candidate_uuid: "9e44efa72e7d4421b284ae74d1cb9634" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47eabc135e9f4237b036097085e729d8"
I20240501 14:08:54.664243 7772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4573be69270c489289af5fc65f67112d" candidate_uuid: "9e44efa72e7d4421b284ae74d1cb9634" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ae5a87ca07b40a788a42734b24f66c6"
I20240501 14:08:54.664371 7904 raft_consensus.cc:3037] T 4573be69270c489289af5fc65f67112d P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:54.664384 7772 raft_consensus.cc:3037] T 4573be69270c489289af5fc65f67112d P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:54.664361 7773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bfcc65d57eb4c3597a761f878a09da2" candidate_uuid: "47eabc135e9f4237b036097085e729d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ae5a87ca07b40a788a42734b24f66c6"
I20240501 14:08:54.664557 7773 raft_consensus.cc:3037] T 0bfcc65d57eb4c3597a761f878a09da2 P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:54.665405 7904 raft_consensus.cc:2454] T 4573be69270c489289af5fc65f67112d P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e44efa72e7d4421b284ae74d1cb9634 in term 1.
I20240501 14:08:54.665405 7772 raft_consensus.cc:2454] T 4573be69270c489289af5fc65f67112d P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e44efa72e7d4421b284ae74d1cb9634 in term 1.
I20240501 14:08:54.665733 8535 leader_election.cc:288] T 4573be69270c489289af5fc65f67112d P 9e44efa72e7d4421b284ae74d1cb9634 [CANDIDATE]: Term 1 election: Requested vote from peers 47eabc135e9f4237b036097085e729d8 (127.7.39.132:42369), 9ae5a87ca07b40a788a42734b24f66c6 (127.7.39.131:34697)
I20240501 14:08:54.665741 7773 raft_consensus.cc:2454] T 0bfcc65d57eb4c3597a761f878a09da2 P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47eabc135e9f4237b036097085e729d8 in term 1.
I20240501 14:08:54.666092 7839 leader_election.cc:302] T 0bfcc65d57eb4c3597a761f878a09da2 P 47eabc135e9f4237b036097085e729d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47eabc135e9f4237b036097085e729d8, 9ae5a87ca07b40a788a42734b24f66c6; no voters:
I20240501 14:08:54.666275 8567 raft_consensus.cc:2781] T 0bfcc65d57eb4c3597a761f878a09da2 P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:54.666261 7642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bfcc65d57eb4c3597a761f878a09da2" candidate_uuid: "47eabc135e9f4237b036097085e729d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e44efa72e7d4421b284ae74d1cb9634"
I20240501 14:08:54.666376 8567 raft_consensus.cc:686] T 0bfcc65d57eb4c3597a761f878a09da2 P 47eabc135e9f4237b036097085e729d8 [term 1 LEADER]: Becoming Leader. State: Replica: 47eabc135e9f4237b036097085e729d8, State: Running, Role: LEADER
I20240501 14:08:54.665791 7577 leader_election.cc:302] T 4573be69270c489289af5fc65f67112d P 9e44efa72e7d4421b284ae74d1cb9634 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ae5a87ca07b40a788a42734b24f66c6, 9e44efa72e7d4421b284ae74d1cb9634; no voters:
I20240501 14:08:54.666399 7642 raft_consensus.cc:3037] T 0bfcc65d57eb4c3597a761f878a09da2 P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:54.666532 8567 consensus_queue.cc:237] T 0bfcc65d57eb4c3597a761f878a09da2 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } }
I20240501 14:08:54.666654 8535 raft_consensus.cc:2781] T 4573be69270c489289af5fc65f67112d P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:54.666733 8535 raft_consensus.cc:686] T 4573be69270c489289af5fc65f67112d P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 LEADER]: Becoming Leader. State: Replica: 9e44efa72e7d4421b284ae74d1cb9634, State: Running, Role: LEADER
I20240501 14:08:54.666855 8535 consensus_queue.cc:237] T 4573be69270c489289af5fc65f67112d P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:08:54.668537 7369 catalog_manager.cc:5484] T 0bfcc65d57eb4c3597a761f878a09da2 P 47eabc135e9f4237b036097085e729d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 47eabc135e9f4237b036097085e729d8 (127.7.39.132). New cstate: current_term: 1 leader_uuid: "47eabc135e9f4237b036097085e729d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } health_report { overall_health: HEALTHY } } }
I20240501 14:08:54.668619 7367 catalog_manager.cc:5484] T 4573be69270c489289af5fc65f67112d P 9e44efa72e7d4421b284ae74d1cb9634 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e44efa72e7d4421b284ae74d1cb9634 (127.7.39.130). New cstate: current_term: 1 leader_uuid: "9e44efa72e7d4421b284ae74d1cb9634" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } health_report { overall_health: UNKNOWN } } }
I20240501 14:08:54.762259 7642 raft_consensus.cc:2454] T 0bfcc65d57eb4c3597a761f878a09da2 P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47eabc135e9f4237b036097085e729d8 in term 1.
W20240501 14:08:54.797317 8605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:54.797588 8605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:54.800798 8605 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 14:08:54.889313 7642 raft_consensus.cc:1261] T 0bfcc65d57eb4c3597a761f878a09da2 P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 FOLLOWER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:54.889767 8567 consensus_queue.cc:1035] T 0bfcc65d57eb4c3597a761f878a09da2 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:54.894218 7904 raft_consensus.cc:1261] T 4573be69270c489289af5fc65f67112d P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Refusing update from remote peer 9e44efa72e7d4421b284ae74d1cb9634: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:54.894830 8535 consensus_queue.cc:1035] T 4573be69270c489289af5fc65f67112d P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:54.896746 7772 raft_consensus.cc:1261] T ae06bed9bd844a05ae2ac69148bf5bad P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Refusing update from remote peer 9e44efa72e7d4421b284ae74d1cb9634: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:54.897223 8535 consensus_queue.cc:1035] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:54.898051 7773 raft_consensus.cc:1261] T 4573be69270c489289af5fc65f67112d P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Refusing update from remote peer 9e44efa72e7d4421b284ae74d1cb9634: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:54.898672 8599 consensus_queue.cc:1035] T 4573be69270c489289af5fc65f67112d P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:54.903697 7511 raft_consensus.cc:1261] T ae06bed9bd844a05ae2ac69148bf5bad P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Refusing update from remote peer 9e44efa72e7d4421b284ae74d1cb9634: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:54.904204 8599 consensus_queue.cc:1035] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:54.908289 7772 raft_consensus.cc:1261] T 0bfcc65d57eb4c3597a761f878a09da2 P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:54.908708 8505 consensus_queue.cc:1035] T 0bfcc65d57eb4c3597a761f878a09da2 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 118.562 ms
time per row: 0.0592808 ms
I20240501 14:08:54.964481 7367 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32870:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:08:54.970710 7489 tablet_service.cc:1467] Processing CreateTablet for tablet ab7c1f81f7524d55bd43065890088eaf (DEFAULT_TABLE table=rebalance_test_table_2 [id=4363cbae8a774bc9b70e97f51629b7b3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:54.971099 7489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab7c1f81f7524d55bd43065890088eaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:54.971796 7884 tablet_service.cc:1467] Processing CreateTablet for tablet 2112bf96dc3343678aa6f46bf2a3b1f2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=4363cbae8a774bc9b70e97f51629b7b3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:54.972062 7884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2112bf96dc3343678aa6f46bf2a3b1f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:54.972002 7620 tablet_service.cc:1467] Processing CreateTablet for tablet 3137e7904ad04fc79d4028c1882e2412 (DEFAULT_TABLE table=rebalance_test_table_2 [id=4363cbae8a774bc9b70e97f51629b7b3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:54.972266 7620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3137e7904ad04fc79d4028c1882e2412. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:54.973547 7883 tablet_service.cc:1467] Processing CreateTablet for tablet ab7c1f81f7524d55bd43065890088eaf (DEFAULT_TABLE table=rebalance_test_table_2 [id=4363cbae8a774bc9b70e97f51629b7b3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:54.973771 7883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab7c1f81f7524d55bd43065890088eaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:54.974263 8584 tablet_bootstrap.cc:492] T 3137e7904ad04fc79d4028c1882e2412 P 9e44efa72e7d4421b284ae74d1cb9634: Bootstrap starting.
I20240501 14:08:54.974479 7752 tablet_service.cc:1467] Processing CreateTablet for tablet 2112bf96dc3343678aa6f46bf2a3b1f2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=4363cbae8a774bc9b70e97f51629b7b3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:54.974759 7752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2112bf96dc3343678aa6f46bf2a3b1f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:54.975001 8584 tablet_bootstrap.cc:654] T 3137e7904ad04fc79d4028c1882e2412 P 9e44efa72e7d4421b284ae74d1cb9634: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:54.975816 7622 tablet_service.cc:1467] Processing CreateTablet for tablet ab7c1f81f7524d55bd43065890088eaf (DEFAULT_TABLE table=rebalance_test_table_2 [id=4363cbae8a774bc9b70e97f51629b7b3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:54.975931 8586 tablet_bootstrap.cc:492] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8: Bootstrap starting.
I20240501 14:08:54.976058 7622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab7c1f81f7524d55bd43065890088eaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:54.976655 7882 tablet_service.cc:1467] Processing CreateTablet for tablet 3137e7904ad04fc79d4028c1882e2412 (DEFAULT_TABLE table=rebalance_test_table_2 [id=4363cbae8a774bc9b70e97f51629b7b3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:54.976800 8586 tablet_bootstrap.cc:654] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:54.976886 7882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3137e7904ad04fc79d4028c1882e2412. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:54.978893 8585 tablet_bootstrap.cc:492] T ab7c1f81f7524d55bd43065890088eaf P ae52f888f5094416a24cb3dffbff03c4: Bootstrap starting.
I20240501 14:08:54.979079 8583 tablet_bootstrap.cc:492] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9ae5a87ca07b40a788a42734b24f66c6: Bootstrap starting.
I20240501 14:08:54.979678 7489 tablet_service.cc:1467] Processing CreateTablet for tablet 2112bf96dc3343678aa6f46bf2a3b1f2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=4363cbae8a774bc9b70e97f51629b7b3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:54.979925 8583 tablet_bootstrap.cc:654] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9ae5a87ca07b40a788a42734b24f66c6: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:54.979959 7489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2112bf96dc3343678aa6f46bf2a3b1f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:54.980502 8585 tablet_bootstrap.cc:654] T ab7c1f81f7524d55bd43065890088eaf P ae52f888f5094416a24cb3dffbff03c4: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:54.981243 8586 tablet_bootstrap.cc:492] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8: No bootstrap required, opened a new log
I20240501 14:08:54.981333 8586 ts_tablet_manager.cc:1367] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:08:54.981565 8584 tablet_bootstrap.cc:492] T 3137e7904ad04fc79d4028c1882e2412 P 9e44efa72e7d4421b284ae74d1cb9634: No bootstrap required, opened a new log
I20240501 14:08:54.979681 7491 tablet_service.cc:1467] Processing CreateTablet for tablet 3137e7904ad04fc79d4028c1882e2412 (DEFAULT_TABLE table=rebalance_test_table_2 [id=4363cbae8a774bc9b70e97f51629b7b3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:54.981650 8584 ts_tablet_manager.cc:1367] T 3137e7904ad04fc79d4028c1882e2412 P 9e44efa72e7d4421b284ae74d1cb9634: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:08:54.981792 7491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3137e7904ad04fc79d4028c1882e2412. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:54.981782 8586 raft_consensus.cc:348] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:08:54.981886 8586 raft_consensus.cc:374] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:54.981920 8586 raft_consensus.cc:729] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47eabc135e9f4237b036097085e729d8, State: Initialized, Role: FOLLOWER
I20240501 14:08:54.982045 8586 consensus_queue.cc:260] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:08:54.982139 8584 raft_consensus.cc:348] T 3137e7904ad04fc79d4028c1882e2412 P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } }
I20240501 14:08:54.982239 8584 raft_consensus.cc:374] T 3137e7904ad04fc79d4028c1882e2412 P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:54.982267 8584 raft_consensus.cc:729] T 3137e7904ad04fc79d4028c1882e2412 P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e44efa72e7d4421b284ae74d1cb9634, State: Initialized, Role: FOLLOWER
I20240501 14:08:54.982319 8586 ts_tablet_manager.cc:1398] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:54.982388 8584 consensus_queue.cc:260] T 3137e7904ad04fc79d4028c1882e2412 P 9e44efa72e7d4421b284ae74d1cb9634 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } }
I20240501 14:08:54.982656 8584 ts_tablet_manager.cc:1398] T 3137e7904ad04fc79d4028c1882e2412 P 9e44efa72e7d4421b284ae74d1cb9634: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:08:54.982825 8583 tablet_bootstrap.cc:492] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9ae5a87ca07b40a788a42734b24f66c6: No bootstrap required, opened a new log
I20240501 14:08:54.982894 8583 ts_tablet_manager.cc:1367] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9ae5a87ca07b40a788a42734b24f66c6: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:08:54.983330 8583 raft_consensus.cc:348] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:08:54.983415 8584 tablet_bootstrap.cc:492] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634: Bootstrap starting.
I20240501 14:08:54.983435 8583 raft_consensus.cc:374] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:54.983467 8583 raft_consensus.cc:729] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ae5a87ca07b40a788a42734b24f66c6, State: Initialized, Role: FOLLOWER
I20240501 14:08:54.983577 8583 consensus_queue.cc:260] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9ae5a87ca07b40a788a42734b24f66c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:08:54.983844 8583 ts_tablet_manager.cc:1398] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9ae5a87ca07b40a788a42734b24f66c6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:54.984331 8584 tablet_bootstrap.cc:654] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:54.984983 8585 tablet_bootstrap.cc:492] T ab7c1f81f7524d55bd43065890088eaf P ae52f888f5094416a24cb3dffbff03c4: No bootstrap required, opened a new log
I20240501 14:08:54.985069 8585 ts_tablet_manager.cc:1367] T ab7c1f81f7524d55bd43065890088eaf P ae52f888f5094416a24cb3dffbff03c4: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20240501 14:08:54.985111 8586 tablet_bootstrap.cc:492] T ab7c1f81f7524d55bd43065890088eaf P 47eabc135e9f4237b036097085e729d8: Bootstrap starting.
I20240501 14:08:54.985440 8585 raft_consensus.cc:348] T ab7c1f81f7524d55bd43065890088eaf P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:08:54.985551 8585 raft_consensus.cc:374] T ab7c1f81f7524d55bd43065890088eaf P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:54.985594 8585 raft_consensus.cc:729] T ab7c1f81f7524d55bd43065890088eaf P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae52f888f5094416a24cb3dffbff03c4, State: Initialized, Role: FOLLOWER
I20240501 14:08:54.985718 8585 consensus_queue.cc:260] T ab7c1f81f7524d55bd43065890088eaf P ae52f888f5094416a24cb3dffbff03c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:08:54.986009 8585 ts_tablet_manager.cc:1398] T ab7c1f81f7524d55bd43065890088eaf P ae52f888f5094416a24cb3dffbff03c4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:54.986021 8586 tablet_bootstrap.cc:654] T ab7c1f81f7524d55bd43065890088eaf P 47eabc135e9f4237b036097085e729d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:54.986186 8585 tablet_bootstrap.cc:492] T 2112bf96dc3343678aa6f46bf2a3b1f2 P ae52f888f5094416a24cb3dffbff03c4: Bootstrap starting.
I20240501 14:08:54.987089 8585 tablet_bootstrap.cc:654] T 2112bf96dc3343678aa6f46bf2a3b1f2 P ae52f888f5094416a24cb3dffbff03c4: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:54.987250 8586 tablet_bootstrap.cc:492] T ab7c1f81f7524d55bd43065890088eaf P 47eabc135e9f4237b036097085e729d8: No bootstrap required, opened a new log
I20240501 14:08:54.987277 8584 tablet_bootstrap.cc:492] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634: No bootstrap required, opened a new log
I20240501 14:08:54.987334 8586 ts_tablet_manager.cc:1367] T ab7c1f81f7524d55bd43065890088eaf P 47eabc135e9f4237b036097085e729d8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:54.987354 8584 ts_tablet_manager.cc:1367] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:08:54.987757 8586 raft_consensus.cc:348] T ab7c1f81f7524d55bd43065890088eaf P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:08:54.987876 8586 raft_consensus.cc:374] T ab7c1f81f7524d55bd43065890088eaf P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:54.987915 8586 raft_consensus.cc:729] T ab7c1f81f7524d55bd43065890088eaf P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47eabc135e9f4237b036097085e729d8, State: Initialized, Role: FOLLOWER
I20240501 14:08:54.987939 8584 raft_consensus.cc:348] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:08:54.988039 8584 raft_consensus.cc:374] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:54.988025 8586 consensus_queue.cc:260] T ab7c1f81f7524d55bd43065890088eaf P 47eabc135e9f4237b036097085e729d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:08:54.988070 8584 raft_consensus.cc:729] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e44efa72e7d4421b284ae74d1cb9634, State: Initialized, Role: FOLLOWER
I20240501 14:08:54.988194 8584 consensus_queue.cc:260] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:08:54.988351 8586 ts_tablet_manager.cc:1398] T ab7c1f81f7524d55bd43065890088eaf P 47eabc135e9f4237b036097085e729d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:54.988485 8584 ts_tablet_manager.cc:1398] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:54.988536 8586 tablet_bootstrap.cc:492] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8: Bootstrap starting.
I20240501 14:08:54.989779 8586 tablet_bootstrap.cc:654] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:54.989874 8585 tablet_bootstrap.cc:492] T 2112bf96dc3343678aa6f46bf2a3b1f2 P ae52f888f5094416a24cb3dffbff03c4: No bootstrap required, opened a new log
I20240501 14:08:54.989957 8585 ts_tablet_manager.cc:1367] T 2112bf96dc3343678aa6f46bf2a3b1f2 P ae52f888f5094416a24cb3dffbff03c4: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:08:54.990406 8585 raft_consensus.cc:348] T 2112bf96dc3343678aa6f46bf2a3b1f2 P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:08:54.990515 8585 raft_consensus.cc:374] T 2112bf96dc3343678aa6f46bf2a3b1f2 P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:54.990553 8585 raft_consensus.cc:729] T 2112bf96dc3343678aa6f46bf2a3b1f2 P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae52f888f5094416a24cb3dffbff03c4, State: Initialized, Role: FOLLOWER
I20240501 14:08:54.990679 8585 consensus_queue.cc:260] T 2112bf96dc3343678aa6f46bf2a3b1f2 P ae52f888f5094416a24cb3dffbff03c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:08:54.990931 8585 ts_tablet_manager.cc:1398] T 2112bf96dc3343678aa6f46bf2a3b1f2 P ae52f888f5094416a24cb3dffbff03c4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:54.991142 8585 tablet_bootstrap.cc:492] T 3137e7904ad04fc79d4028c1882e2412 P ae52f888f5094416a24cb3dffbff03c4: Bootstrap starting.
I20240501 14:08:54.991638 8586 tablet_bootstrap.cc:492] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8: No bootstrap required, opened a new log
I20240501 14:08:54.991747 8586 ts_tablet_manager.cc:1367] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:08:54.992142 8585 tablet_bootstrap.cc:654] T 3137e7904ad04fc79d4028c1882e2412 P ae52f888f5094416a24cb3dffbff03c4: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:54.992185 8586 raft_consensus.cc:348] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } }
I20240501 14:08:54.992290 8586 raft_consensus.cc:374] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:54.992334 8586 raft_consensus.cc:729] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47eabc135e9f4237b036097085e729d8, State: Initialized, Role: FOLLOWER
I20240501 14:08:54.992449 8586 consensus_queue.cc:260] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } }
I20240501 14:08:54.992765 8586 ts_tablet_manager.cc:1398] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:54.993347 8585 tablet_bootstrap.cc:492] T 3137e7904ad04fc79d4028c1882e2412 P ae52f888f5094416a24cb3dffbff03c4: No bootstrap required, opened a new log
I20240501 14:08:54.993433 8585 ts_tablet_manager.cc:1367] T 3137e7904ad04fc79d4028c1882e2412 P ae52f888f5094416a24cb3dffbff03c4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:54.993870 8585 raft_consensus.cc:348] T 3137e7904ad04fc79d4028c1882e2412 P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } }
I20240501 14:08:54.993989 8585 raft_consensus.cc:374] T 3137e7904ad04fc79d4028c1882e2412 P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:54.994035 8585 raft_consensus.cc:729] T 3137e7904ad04fc79d4028c1882e2412 P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae52f888f5094416a24cb3dffbff03c4, State: Initialized, Role: FOLLOWER
I20240501 14:08:54.994176 8585 consensus_queue.cc:260] T 3137e7904ad04fc79d4028c1882e2412 P ae52f888f5094416a24cb3dffbff03c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } }
I20240501 14:08:54.994462 8585 ts_tablet_manager.cc:1398] T 3137e7904ad04fc79d4028c1882e2412 P ae52f888f5094416a24cb3dffbff03c4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:55.020998 8599 raft_consensus.cc:482] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:55.021133 8599 raft_consensus.cc:504] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:08:55.021544 8599 leader_election.cc:288] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083), 47eabc135e9f4237b036097085e729d8 (127.7.39.132:42369)
I20240501 14:08:55.021997 7904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7c1f81f7524d55bd43065890088eaf" candidate_uuid: "9e44efa72e7d4421b284ae74d1cb9634" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47eabc135e9f4237b036097085e729d8" is_pre_election: true
I20240501 14:08:55.022003 7511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7c1f81f7524d55bd43065890088eaf" candidate_uuid: "9e44efa72e7d4421b284ae74d1cb9634" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae52f888f5094416a24cb3dffbff03c4" is_pre_election: true
I20240501 14:08:55.022142 7904 raft_consensus.cc:2454] T ab7c1f81f7524d55bd43065890088eaf P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e44efa72e7d4421b284ae74d1cb9634 in term 0.
I20240501 14:08:55.022186 7511 raft_consensus.cc:2454] T ab7c1f81f7524d55bd43065890088eaf P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e44efa72e7d4421b284ae74d1cb9634 in term 0.
I20240501 14:08:55.022537 7577 leader_election.cc:302] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e44efa72e7d4421b284ae74d1cb9634, ae52f888f5094416a24cb3dffbff03c4; no voters:
I20240501 14:08:55.022729 8599 raft_consensus.cc:2781] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:55.022801 8599 raft_consensus.cc:482] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:55.022861 8599 raft_consensus.cc:3037] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:55.068763 8599 raft_consensus.cc:504] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:08:55.069306 8599 leader_election.cc:288] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 [CANDIDATE]: Term 1 election: Requested vote from peers ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083), 47eabc135e9f4237b036097085e729d8 (127.7.39.132:42369)
I20240501 14:08:55.069778 7904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7c1f81f7524d55bd43065890088eaf" candidate_uuid: "9e44efa72e7d4421b284ae74d1cb9634" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47eabc135e9f4237b036097085e729d8"
I20240501 14:08:55.069800 7511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7c1f81f7524d55bd43065890088eaf" candidate_uuid: "9e44efa72e7d4421b284ae74d1cb9634" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae52f888f5094416a24cb3dffbff03c4"
I20240501 14:08:55.069921 7904 raft_consensus.cc:3037] T ab7c1f81f7524d55bd43065890088eaf P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:55.069960 7511 raft_consensus.cc:3037] T ab7c1f81f7524d55bd43065890088eaf P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:55.070843 7904 raft_consensus.cc:2454] T ab7c1f81f7524d55bd43065890088eaf P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e44efa72e7d4421b284ae74d1cb9634 in term 1.
I20240501 14:08:55.071228 7511 raft_consensus.cc:2454] T ab7c1f81f7524d55bd43065890088eaf P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e44efa72e7d4421b284ae74d1cb9634 in term 1.
I20240501 14:08:55.071540 7576 leader_election.cc:302] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47eabc135e9f4237b036097085e729d8, 9e44efa72e7d4421b284ae74d1cb9634; no voters:
I20240501 14:08:55.071779 8599 raft_consensus.cc:2781] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:55.071942 8599 raft_consensus.cc:686] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 LEADER]: Becoming Leader. State: Replica: 9e44efa72e7d4421b284ae74d1cb9634, State: Running, Role: LEADER
I20240501 14:08:55.072144 8599 consensus_queue.cc:237] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:08:55.074182 7367 catalog_manager.cc:5484] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e44efa72e7d4421b284ae74d1cb9634 (127.7.39.130). New cstate: current_term: 1 leader_uuid: "9e44efa72e7d4421b284ae74d1cb9634" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } health_report { overall_health: HEALTHY } } }
I20240501 14:08:55.130180 8642 raft_consensus.cc:482] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:55.130338 8642 raft_consensus.cc:504] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:08:55.130754 8642 leader_election.cc:288] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083), 9ae5a87ca07b40a788a42734b24f66c6 (127.7.39.131:34697)
I20240501 14:08:55.131337 7511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2112bf96dc3343678aa6f46bf2a3b1f2" candidate_uuid: "47eabc135e9f4237b036097085e729d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae52f888f5094416a24cb3dffbff03c4" is_pre_election: true
I20240501 14:08:55.131393 7772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2112bf96dc3343678aa6f46bf2a3b1f2" candidate_uuid: "47eabc135e9f4237b036097085e729d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ae5a87ca07b40a788a42734b24f66c6" is_pre_election: true
I20240501 14:08:55.131491 7511 raft_consensus.cc:2454] T 2112bf96dc3343678aa6f46bf2a3b1f2 P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47eabc135e9f4237b036097085e729d8 in term 0.
I20240501 14:08:55.131589 7772 raft_consensus.cc:2454] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47eabc135e9f4237b036097085e729d8 in term 0.
I20240501 14:08:55.131834 7839 leader_election.cc:302] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47eabc135e9f4237b036097085e729d8, ae52f888f5094416a24cb3dffbff03c4; no voters:
I20240501 14:08:55.132035 8642 raft_consensus.cc:2781] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:55.132098 8642 raft_consensus.cc:482] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:55.132146 8642 raft_consensus.cc:3037] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:55.169808 8642 raft_consensus.cc:504] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:08:55.170433 8642 leader_election.cc:288] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [CANDIDATE]: Term 1 election: Requested vote from peers ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083), 9ae5a87ca07b40a788a42734b24f66c6 (127.7.39.131:34697)
I20240501 14:08:55.170948 7511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2112bf96dc3343678aa6f46bf2a3b1f2" candidate_uuid: "47eabc135e9f4237b036097085e729d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae52f888f5094416a24cb3dffbff03c4"
I20240501 14:08:55.171013 7772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2112bf96dc3343678aa6f46bf2a3b1f2" candidate_uuid: "47eabc135e9f4237b036097085e729d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ae5a87ca07b40a788a42734b24f66c6"
I20240501 14:08:55.171133 7511 raft_consensus.cc:3037] T 2112bf96dc3343678aa6f46bf2a3b1f2 P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:55.171204 7772 raft_consensus.cc:3037] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:55.172070 7772 raft_consensus.cc:2454] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47eabc135e9f4237b036097085e729d8 in term 1.
I20240501 14:08:55.172395 7511 raft_consensus.cc:2454] T 2112bf96dc3343678aa6f46bf2a3b1f2 P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47eabc135e9f4237b036097085e729d8 in term 1.
I20240501 14:08:55.172444 7839 leader_election.cc:302] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47eabc135e9f4237b036097085e729d8, 9ae5a87ca07b40a788a42734b24f66c6; no voters:
I20240501 14:08:55.172626 8642 raft_consensus.cc:2781] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:55.172708 8642 raft_consensus.cc:686] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [term 1 LEADER]: Becoming Leader. State: Replica: 47eabc135e9f4237b036097085e729d8, State: Running, Role: LEADER
I20240501 14:08:55.172863 8642 consensus_queue.cc:237] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:08:55.174700 7367 catalog_manager.cc:5484] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 47eabc135e9f4237b036097085e729d8 (127.7.39.132). New cstate: current_term: 1 leader_uuid: "47eabc135e9f4237b036097085e729d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } health_report { overall_health: UNKNOWN } } }
I20240501 14:08:55.364858 8501 raft_consensus.cc:482] T 3137e7904ad04fc79d4028c1882e2412 P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:55.365025 8501 raft_consensus.cc:504] T 3137e7904ad04fc79d4028c1882e2412 P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } }
I20240501 14:08:55.365494 8501 leader_election.cc:288] T 3137e7904ad04fc79d4028c1882e2412 P ae52f888f5094416a24cb3dffbff03c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e44efa72e7d4421b284ae74d1cb9634 (127.7.39.130:46433), 47eabc135e9f4237b036097085e729d8 (127.7.39.132:42369)
I20240501 14:08:55.365974 7904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3137e7904ad04fc79d4028c1882e2412" candidate_uuid: "ae52f888f5094416a24cb3dffbff03c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47eabc135e9f4237b036097085e729d8" is_pre_election: true
I20240501 14:08:55.365975 7642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3137e7904ad04fc79d4028c1882e2412" candidate_uuid: "ae52f888f5094416a24cb3dffbff03c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e44efa72e7d4421b284ae74d1cb9634" is_pre_election: true
I20240501 14:08:55.366117 7642 raft_consensus.cc:2454] T 3137e7904ad04fc79d4028c1882e2412 P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae52f888f5094416a24cb3dffbff03c4 in term 0.
I20240501 14:08:55.366127 7904 raft_consensus.cc:2454] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae52f888f5094416a24cb3dffbff03c4 in term 0.
I20240501 14:08:55.366500 7445 leader_election.cc:302] T 3137e7904ad04fc79d4028c1882e2412 P ae52f888f5094416a24cb3dffbff03c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47eabc135e9f4237b036097085e729d8, ae52f888f5094416a24cb3dffbff03c4; no voters:
I20240501 14:08:55.366680 8501 raft_consensus.cc:2781] T 3137e7904ad04fc79d4028c1882e2412 P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:55.366755 8501 raft_consensus.cc:482] T 3137e7904ad04fc79d4028c1882e2412 P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:55.366793 8501 raft_consensus.cc:3037] T 3137e7904ad04fc79d4028c1882e2412 P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:55.370492 8501 raft_consensus.cc:504] T 3137e7904ad04fc79d4028c1882e2412 P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } }
I20240501 14:08:55.370884 8501 leader_election.cc:288] T 3137e7904ad04fc79d4028c1882e2412 P ae52f888f5094416a24cb3dffbff03c4 [CANDIDATE]: Term 1 election: Requested vote from peers 9e44efa72e7d4421b284ae74d1cb9634 (127.7.39.130:46433), 47eabc135e9f4237b036097085e729d8 (127.7.39.132:42369)
I20240501 14:08:55.371318 7642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3137e7904ad04fc79d4028c1882e2412" candidate_uuid: "ae52f888f5094416a24cb3dffbff03c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e44efa72e7d4421b284ae74d1cb9634"
I20240501 14:08:55.371361 7904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3137e7904ad04fc79d4028c1882e2412" candidate_uuid: "ae52f888f5094416a24cb3dffbff03c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47eabc135e9f4237b036097085e729d8"
I20240501 14:08:55.371474 7642 raft_consensus.cc:3037] T 3137e7904ad04fc79d4028c1882e2412 P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:55.371492 7904 raft_consensus.cc:3037] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:55.372283 7904 raft_consensus.cc:2454] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae52f888f5094416a24cb3dffbff03c4 in term 1.
I20240501 14:08:55.372611 7445 leader_election.cc:302] T 3137e7904ad04fc79d4028c1882e2412 P ae52f888f5094416a24cb3dffbff03c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47eabc135e9f4237b036097085e729d8, ae52f888f5094416a24cb3dffbff03c4; no voters:
I20240501 14:08:55.372722 7642 raft_consensus.cc:2454] T 3137e7904ad04fc79d4028c1882e2412 P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae52f888f5094416a24cb3dffbff03c4 in term 1.
I20240501 14:08:55.372778 8501 raft_consensus.cc:2781] T 3137e7904ad04fc79d4028c1882e2412 P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:55.372962 8501 raft_consensus.cc:686] T 3137e7904ad04fc79d4028c1882e2412 P ae52f888f5094416a24cb3dffbff03c4 [term 1 LEADER]: Becoming Leader. State: Replica: ae52f888f5094416a24cb3dffbff03c4, State: Running, Role: LEADER
I20240501 14:08:55.373124 8501 consensus_queue.cc:237] T 3137e7904ad04fc79d4028c1882e2412 P ae52f888f5094416a24cb3dffbff03c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } }
I20240501 14:08:55.375025 7367 catalog_manager.cc:5484] T 3137e7904ad04fc79d4028c1882e2412 P ae52f888f5094416a24cb3dffbff03c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129). New cstate: current_term: 1 leader_uuid: "ae52f888f5094416a24cb3dffbff03c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } health_report { overall_health: UNKNOWN } } }
I20240501 14:08:55.535362 8599 consensus_queue.cc:1035] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:55.539147 8535 consensus_queue.cc:1035] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:08:55.551131 8656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:55.551385 8656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:55.554716 8656 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240501 14:08:55.638566 7772 raft_consensus.cc:1261] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:55.639384 7903 raft_consensus.cc:1261] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Refusing update from remote peer ae52f888f5094416a24cb3dffbff03c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:55.639918 8501 consensus_queue.cc:1035] T 3137e7904ad04fc79d4028c1882e2412 P ae52f888f5094416a24cb3dffbff03c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:55.640658 8642 consensus_queue.cc:1035] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:55.645922 7511 raft_consensus.cc:1261] T 2112bf96dc3343678aa6f46bf2a3b1f2 P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:55.648495 7642 raft_consensus.cc:1261] T 3137e7904ad04fc79d4028c1882e2412 P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 FOLLOWER]: Refusing update from remote peer ae52f888f5094416a24cb3dffbff03c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:55.648542 8703 consensus_queue.cc:1035] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:55.650002 8501 consensus_queue.cc:1035] T 3137e7904ad04fc79d4028c1882e2412 P ae52f888f5094416a24cb3dffbff03c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 110.354 ms
time per row: 0.0551768 ms
I20240501 14:08:55.709000 7368 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32870:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:08:55.714371 7752 tablet_service.cc:1467] Processing CreateTablet for tablet 2880045de45743bdb5358a17d93b2245 (DEFAULT_TABLE table=rebalance_test_table_3 [id=1efd7e27dfbf4a88a4c2c2d4d7ebfe4f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:55.714744 7752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2880045de45743bdb5358a17d93b2245. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:55.714680 7491 tablet_service.cc:1467] Processing CreateTablet for tablet 2880045de45743bdb5358a17d93b2245 (DEFAULT_TABLE table=rebalance_test_table_3 [id=1efd7e27dfbf4a88a4c2c2d4d7ebfe4f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:55.714972 7491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2880045de45743bdb5358a17d93b2245. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:55.715354 7622 tablet_service.cc:1467] Processing CreateTablet for tablet 2880045de45743bdb5358a17d93b2245 (DEFAULT_TABLE table=rebalance_test_table_3 [id=1efd7e27dfbf4a88a4c2c2d4d7ebfe4f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:55.715517 7884 tablet_service.cc:1467] Processing CreateTablet for tablet 4d282d79e1664f7ea8912e0c5166eec3 (DEFAULT_TABLE table=rebalance_test_table_3 [id=1efd7e27dfbf4a88a4c2c2d4d7ebfe4f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:55.715657 7622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2880045de45743bdb5358a17d93b2245. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:55.715770 7884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d282d79e1664f7ea8912e0c5166eec3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:55.717355 7882 tablet_service.cc:1467] Processing CreateTablet for tablet 6841e1b32b86454f9e40b64c141a2a6b (DEFAULT_TABLE table=rebalance_test_table_3 [id=1efd7e27dfbf4a88a4c2c2d4d7ebfe4f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:55.717605 7882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6841e1b32b86454f9e40b64c141a2a6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:55.717949 7752 tablet_service.cc:1467] Processing CreateTablet for tablet 6841e1b32b86454f9e40b64c141a2a6b (DEFAULT_TABLE table=rebalance_test_table_3 [id=1efd7e27dfbf4a88a4c2c2d4d7ebfe4f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:55.718175 7752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6841e1b32b86454f9e40b64c141a2a6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:55.718417 7622 tablet_service.cc:1467] Processing CreateTablet for tablet 6841e1b32b86454f9e40b64c141a2a6b (DEFAULT_TABLE table=rebalance_test_table_3 [id=1efd7e27dfbf4a88a4c2c2d4d7ebfe4f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:55.718538 8720 tablet_bootstrap.cc:492] T 2880045de45743bdb5358a17d93b2245 P 9e44efa72e7d4421b284ae74d1cb9634: Bootstrap starting.
I20240501 14:08:55.718712 7622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6841e1b32b86454f9e40b64c141a2a6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:55.714371 7753 tablet_service.cc:1467] Processing CreateTablet for tablet 4d282d79e1664f7ea8912e0c5166eec3 (DEFAULT_TABLE table=rebalance_test_table_3 [id=1efd7e27dfbf4a88a4c2c2d4d7ebfe4f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:55.719583 8720 tablet_bootstrap.cc:654] T 2880045de45743bdb5358a17d93b2245 P 9e44efa72e7d4421b284ae74d1cb9634: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:55.719702 7753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d282d79e1664f7ea8912e0c5166eec3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:55.721204 8721 tablet_bootstrap.cc:492] T 4d282d79e1664f7ea8912e0c5166eec3 P 47eabc135e9f4237b036097085e729d8: Bootstrap starting.
I20240501 14:08:55.721339 8720 tablet_bootstrap.cc:492] T 2880045de45743bdb5358a17d93b2245 P 9e44efa72e7d4421b284ae74d1cb9634: No bootstrap required, opened a new log
I20240501 14:08:55.721469 8720 ts_tablet_manager.cc:1367] T 2880045de45743bdb5358a17d93b2245 P 9e44efa72e7d4421b284ae74d1cb9634: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:08:55.721740 7491 tablet_service.cc:1467] Processing CreateTablet for tablet 4d282d79e1664f7ea8912e0c5166eec3 (DEFAULT_TABLE table=rebalance_test_table_3 [id=1efd7e27dfbf4a88a4c2c2d4d7ebfe4f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:55.721982 7491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d282d79e1664f7ea8912e0c5166eec3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:55.721961 8720 raft_consensus.cc:348] T 2880045de45743bdb5358a17d93b2245 P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:08:55.722105 8720 raft_consensus.cc:374] T 2880045de45743bdb5358a17d93b2245 P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:55.722115 8721 tablet_bootstrap.cc:654] T 4d282d79e1664f7ea8912e0c5166eec3 P 47eabc135e9f4237b036097085e729d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:55.722189 8720 raft_consensus.cc:729] T 2880045de45743bdb5358a17d93b2245 P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e44efa72e7d4421b284ae74d1cb9634, State: Initialized, Role: FOLLOWER
I20240501 14:08:55.722362 8720 consensus_queue.cc:260] T 2880045de45743bdb5358a17d93b2245 P 9e44efa72e7d4421b284ae74d1cb9634 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:08:55.722749 8720 ts_tablet_manager.cc:1398] T 2880045de45743bdb5358a17d93b2245 P 9e44efa72e7d4421b284ae74d1cb9634: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:55.722914 8720 tablet_bootstrap.cc:492] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634: Bootstrap starting.
I20240501 14:08:55.723268 8721 tablet_bootstrap.cc:492] T 4d282d79e1664f7ea8912e0c5166eec3 P 47eabc135e9f4237b036097085e729d8: No bootstrap required, opened a new log
I20240501 14:08:55.724217 8721 ts_tablet_manager.cc:1367] T 4d282d79e1664f7ea8912e0c5166eec3 P 47eabc135e9f4237b036097085e729d8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:08:55.724503 8720 tablet_bootstrap.cc:654] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:55.724913 8719 tablet_bootstrap.cc:492] T 2880045de45743bdb5358a17d93b2245 P ae52f888f5094416a24cb3dffbff03c4: Bootstrap starting.
I20240501 14:08:55.725137 8718 tablet_bootstrap.cc:492] T 2880045de45743bdb5358a17d93b2245 P 9ae5a87ca07b40a788a42734b24f66c6: Bootstrap starting.
I20240501 14:08:55.724797 8721 raft_consensus.cc:348] T 4d282d79e1664f7ea8912e0c5166eec3 P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } }
I20240501 14:08:55.725628 8721 raft_consensus.cc:374] T 4d282d79e1664f7ea8912e0c5166eec3 P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:55.725701 8721 raft_consensus.cc:729] T 4d282d79e1664f7ea8912e0c5166eec3 P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47eabc135e9f4237b036097085e729d8, State: Initialized, Role: FOLLOWER
I20240501 14:08:55.725867 8721 consensus_queue.cc:260] T 4d282d79e1664f7ea8912e0c5166eec3 P 47eabc135e9f4237b036097085e729d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } }
I20240501 14:08:55.726049 8720 tablet_bootstrap.cc:492] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634: No bootstrap required, opened a new log
I20240501 14:08:55.726176 8721 ts_tablet_manager.cc:1398] T 4d282d79e1664f7ea8912e0c5166eec3 P 47eabc135e9f4237b036097085e729d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:55.726176 8720 ts_tablet_manager.cc:1367] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:08:55.726385 8721 tablet_bootstrap.cc:492] T 6841e1b32b86454f9e40b64c141a2a6b P 47eabc135e9f4237b036097085e729d8: Bootstrap starting.
I20240501 14:08:55.726601 8720 raft_consensus.cc:348] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } }
I20240501 14:08:55.726768 8720 raft_consensus.cc:374] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:55.726861 8720 raft_consensus.cc:729] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e44efa72e7d4421b284ae74d1cb9634, State: Initialized, Role: FOLLOWER
I20240501 14:08:55.726933 8719 tablet_bootstrap.cc:654] T 2880045de45743bdb5358a17d93b2245 P ae52f888f5094416a24cb3dffbff03c4: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:55.727087 8720 consensus_queue.cc:260] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } }
I20240501 14:08:55.727376 8721 tablet_bootstrap.cc:654] T 6841e1b32b86454f9e40b64c141a2a6b P 47eabc135e9f4237b036097085e729d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:55.727522 8720 ts_tablet_manager.cc:1398] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:55.727757 8718 tablet_bootstrap.cc:654] T 2880045de45743bdb5358a17d93b2245 P 9ae5a87ca07b40a788a42734b24f66c6: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:55.728502 8721 tablet_bootstrap.cc:492] T 6841e1b32b86454f9e40b64c141a2a6b P 47eabc135e9f4237b036097085e729d8: No bootstrap required, opened a new log
I20240501 14:08:55.728583 8721 ts_tablet_manager.cc:1367] T 6841e1b32b86454f9e40b64c141a2a6b P 47eabc135e9f4237b036097085e729d8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:55.728695 8719 tablet_bootstrap.cc:492] T 2880045de45743bdb5358a17d93b2245 P ae52f888f5094416a24cb3dffbff03c4: No bootstrap required, opened a new log
I20240501 14:08:55.728772 8719 ts_tablet_manager.cc:1367] T 2880045de45743bdb5358a17d93b2245 P ae52f888f5094416a24cb3dffbff03c4: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:08:55.729048 8721 raft_consensus.cc:348] T 6841e1b32b86454f9e40b64c141a2a6b P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } }
I20240501 14:08:55.729154 8721 raft_consensus.cc:374] T 6841e1b32b86454f9e40b64c141a2a6b P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:55.729189 8721 raft_consensus.cc:729] T 6841e1b32b86454f9e40b64c141a2a6b P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47eabc135e9f4237b036097085e729d8, State: Initialized, Role: FOLLOWER
I20240501 14:08:55.729259 8719 raft_consensus.cc:348] T 2880045de45743bdb5358a17d93b2245 P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:08:55.729302 8721 consensus_queue.cc:260] T 6841e1b32b86454f9e40b64c141a2a6b P 47eabc135e9f4237b036097085e729d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } }
I20240501 14:08:55.729367 8719 raft_consensus.cc:374] T 2880045de45743bdb5358a17d93b2245 P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:55.729400 8719 raft_consensus.cc:729] T 2880045de45743bdb5358a17d93b2245 P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae52f888f5094416a24cb3dffbff03c4, State: Initialized, Role: FOLLOWER
I20240501 14:08:55.729609 8721 ts_tablet_manager.cc:1398] T 6841e1b32b86454f9e40b64c141a2a6b P 47eabc135e9f4237b036097085e729d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:55.729611 8719 consensus_queue.cc:260] T 2880045de45743bdb5358a17d93b2245 P ae52f888f5094416a24cb3dffbff03c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:08:55.730178 8718 tablet_bootstrap.cc:492] T 2880045de45743bdb5358a17d93b2245 P 9ae5a87ca07b40a788a42734b24f66c6: No bootstrap required, opened a new log
I20240501 14:08:55.730290 8718 ts_tablet_manager.cc:1367] T 2880045de45743bdb5358a17d93b2245 P 9ae5a87ca07b40a788a42734b24f66c6: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:08:55.730737 8718 raft_consensus.cc:348] T 2880045de45743bdb5358a17d93b2245 P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:08:55.730893 8718 raft_consensus.cc:374] T 2880045de45743bdb5358a17d93b2245 P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:55.730943 8718 raft_consensus.cc:729] T 2880045de45743bdb5358a17d93b2245 P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ae5a87ca07b40a788a42734b24f66c6, State: Initialized, Role: FOLLOWER
I20240501 14:08:55.731256 8718 consensus_queue.cc:260] T 2880045de45743bdb5358a17d93b2245 P 9ae5a87ca07b40a788a42734b24f66c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:08:55.731678 8718 ts_tablet_manager.cc:1398] T 2880045de45743bdb5358a17d93b2245 P 9ae5a87ca07b40a788a42734b24f66c6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:55.731905 8718 tablet_bootstrap.cc:492] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6: Bootstrap starting.
I20240501 14:08:55.731894 8719 ts_tablet_manager.cc:1398] T 2880045de45743bdb5358a17d93b2245 P ae52f888f5094416a24cb3dffbff03c4: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:08:55.732074 8719 tablet_bootstrap.cc:492] T 4d282d79e1664f7ea8912e0c5166eec3 P ae52f888f5094416a24cb3dffbff03c4: Bootstrap starting.
I20240501 14:08:55.732842 8718 tablet_bootstrap.cc:654] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:55.733175 8719 tablet_bootstrap.cc:654] T 4d282d79e1664f7ea8912e0c5166eec3 P ae52f888f5094416a24cb3dffbff03c4: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:55.734026 8718 tablet_bootstrap.cc:492] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6: No bootstrap required, opened a new log
I20240501 14:08:55.734133 8718 ts_tablet_manager.cc:1367] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:55.734601 8718 raft_consensus.cc:348] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } }
I20240501 14:08:55.734709 8718 raft_consensus.cc:374] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:55.734748 8718 raft_consensus.cc:729] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ae5a87ca07b40a788a42734b24f66c6, State: Initialized, Role: FOLLOWER
I20240501 14:08:55.734882 8718 consensus_queue.cc:260] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } }
I20240501 14:08:55.737109 8718 ts_tablet_manager.cc:1398] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:08:55.737273 8718 tablet_bootstrap.cc:492] T 4d282d79e1664f7ea8912e0c5166eec3 P 9ae5a87ca07b40a788a42734b24f66c6: Bootstrap starting.
I20240501 14:08:55.738103 8718 tablet_bootstrap.cc:654] T 4d282d79e1664f7ea8912e0c5166eec3 P 9ae5a87ca07b40a788a42734b24f66c6: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:55.739094 8719 tablet_bootstrap.cc:492] T 4d282d79e1664f7ea8912e0c5166eec3 P ae52f888f5094416a24cb3dffbff03c4: No bootstrap required, opened a new log
I20240501 14:08:55.739249 8719 ts_tablet_manager.cc:1367] T 4d282d79e1664f7ea8912e0c5166eec3 P ae52f888f5094416a24cb3dffbff03c4: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:08:55.739322 8718 tablet_bootstrap.cc:492] T 4d282d79e1664f7ea8912e0c5166eec3 P 9ae5a87ca07b40a788a42734b24f66c6: No bootstrap required, opened a new log
I20240501 14:08:55.739406 8718 ts_tablet_manager.cc:1367] T 4d282d79e1664f7ea8912e0c5166eec3 P 9ae5a87ca07b40a788a42734b24f66c6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:55.739895 8719 raft_consensus.cc:348] T 4d282d79e1664f7ea8912e0c5166eec3 P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } }
I20240501 14:08:55.739840 8718 raft_consensus.cc:348] T 4d282d79e1664f7ea8912e0c5166eec3 P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } }
I20240501 14:08:55.740077 8718 raft_consensus.cc:374] T 4d282d79e1664f7ea8912e0c5166eec3 P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:55.740121 8718 raft_consensus.cc:729] T 4d282d79e1664f7ea8912e0c5166eec3 P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ae5a87ca07b40a788a42734b24f66c6, State: Initialized, Role: FOLLOWER
I20240501 14:08:55.740252 8718 consensus_queue.cc:260] T 4d282d79e1664f7ea8912e0c5166eec3 P 9ae5a87ca07b40a788a42734b24f66c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } }
I20240501 14:08:55.740485 8718 ts_tablet_manager.cc:1398] T 4d282d79e1664f7ea8912e0c5166eec3 P 9ae5a87ca07b40a788a42734b24f66c6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:55.740108 8719 raft_consensus.cc:374] T 4d282d79e1664f7ea8912e0c5166eec3 P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:55.741199 8719 raft_consensus.cc:729] T 4d282d79e1664f7ea8912e0c5166eec3 P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae52f888f5094416a24cb3dffbff03c4, State: Initialized, Role: FOLLOWER
I20240501 14:08:55.741372 8719 consensus_queue.cc:260] T 4d282d79e1664f7ea8912e0c5166eec3 P ae52f888f5094416a24cb3dffbff03c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } }
I20240501 14:08:55.741750 8719 ts_tablet_manager.cc:1398] T 4d282d79e1664f7ea8912e0c5166eec3 P ae52f888f5094416a24cb3dffbff03c4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:55.811738 8709 raft_consensus.cc:482] T 2880045de45743bdb5358a17d93b2245 P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:55.811901 8709 raft_consensus.cc:504] T 2880045de45743bdb5358a17d93b2245 P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:08:55.813014 8709 leader_election.cc:288] T 2880045de45743bdb5358a17d93b2245 P 9ae5a87ca07b40a788a42734b24f66c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083), 9e44efa72e7d4421b284ae74d1cb9634 (127.7.39.130:46433)
I20240501 14:08:55.817060 7642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2880045de45743bdb5358a17d93b2245" candidate_uuid: "9ae5a87ca07b40a788a42734b24f66c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e44efa72e7d4421b284ae74d1cb9634" is_pre_election: true
I20240501 14:08:55.817133 7511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2880045de45743bdb5358a17d93b2245" candidate_uuid: "9ae5a87ca07b40a788a42734b24f66c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae52f888f5094416a24cb3dffbff03c4" is_pre_election: true
I20240501 14:08:55.817235 7642 raft_consensus.cc:2454] T 2880045de45743bdb5358a17d93b2245 P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ae5a87ca07b40a788a42734b24f66c6 in term 0.
I20240501 14:08:55.817268 7511 raft_consensus.cc:2454] T 2880045de45743bdb5358a17d93b2245 P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ae5a87ca07b40a788a42734b24f66c6 in term 0.
I20240501 14:08:55.817600 7708 leader_election.cc:302] T 2880045de45743bdb5358a17d93b2245 P 9ae5a87ca07b40a788a42734b24f66c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ae5a87ca07b40a788a42734b24f66c6, ae52f888f5094416a24cb3dffbff03c4; no voters:
I20240501 14:08:55.817869 8709 raft_consensus.cc:2781] T 2880045de45743bdb5358a17d93b2245 P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:55.817931 8709 raft_consensus.cc:482] T 2880045de45743bdb5358a17d93b2245 P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:55.817986 8709 raft_consensus.cc:3037] T 2880045de45743bdb5358a17d93b2245 P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:55.837700 8700 raft_consensus.cc:482] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:55.837859 8700 raft_consensus.cc:504] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } }
I20240501 14:08:55.838414 8700 leader_election.cc:288] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e44efa72e7d4421b284ae74d1cb9634 (127.7.39.130:46433), 47eabc135e9f4237b036097085e729d8 (127.7.39.132:42369)
I20240501 14:08:55.838898 7642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6841e1b32b86454f9e40b64c141a2a6b" candidate_uuid: "9ae5a87ca07b40a788a42734b24f66c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e44efa72e7d4421b284ae74d1cb9634" is_pre_election: true
I20240501 14:08:55.839105 7642 raft_consensus.cc:2454] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ae5a87ca07b40a788a42734b24f66c6 in term 0.
I20240501 14:08:55.839387 7708 leader_election.cc:302] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ae5a87ca07b40a788a42734b24f66c6, 9e44efa72e7d4421b284ae74d1cb9634; no voters:
I20240501 14:08:55.839556 8700 raft_consensus.cc:2781] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:55.839622 8700 raft_consensus.cc:482] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:55.839672 8700 raft_consensus.cc:3037] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:55.841991 7903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6841e1b32b86454f9e40b64c141a2a6b" candidate_uuid: "9ae5a87ca07b40a788a42734b24f66c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47eabc135e9f4237b036097085e729d8" is_pre_election: true
I20240501 14:08:55.842144 7903 raft_consensus.cc:2454] T 6841e1b32b86454f9e40b64c141a2a6b P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ae5a87ca07b40a788a42734b24f66c6 in term 0.
I20240501 14:08:55.848277 8703 raft_consensus.cc:482] T 4d282d79e1664f7ea8912e0c5166eec3 P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:55.848398 8703 raft_consensus.cc:504] T 4d282d79e1664f7ea8912e0c5166eec3 P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } }
I20240501 14:08:55.848848 8703 leader_election.cc:288] T 4d282d79e1664f7ea8912e0c5166eec3 P 47eabc135e9f4237b036097085e729d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ae5a87ca07b40a788a42734b24f66c6 (127.7.39.131:34697), ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083)
I20240501 14:08:55.849336 7772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d282d79e1664f7ea8912e0c5166eec3" candidate_uuid: "47eabc135e9f4237b036097085e729d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ae5a87ca07b40a788a42734b24f66c6" is_pre_election: true
I20240501 14:08:55.849385 7511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d282d79e1664f7ea8912e0c5166eec3" candidate_uuid: "47eabc135e9f4237b036097085e729d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae52f888f5094416a24cb3dffbff03c4" is_pre_election: true
I20240501 14:08:55.849474 7772 raft_consensus.cc:2454] T 4d282d79e1664f7ea8912e0c5166eec3 P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47eabc135e9f4237b036097085e729d8 in term 0.
I20240501 14:08:55.849531 7511 raft_consensus.cc:2454] T 4d282d79e1664f7ea8912e0c5166eec3 P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47eabc135e9f4237b036097085e729d8 in term 0.
I20240501 14:08:55.849829 7839 leader_election.cc:302] T 4d282d79e1664f7ea8912e0c5166eec3 P 47eabc135e9f4237b036097085e729d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47eabc135e9f4237b036097085e729d8, 9ae5a87ca07b40a788a42734b24f66c6; no voters:
I20240501 14:08:55.850005 8703 raft_consensus.cc:2781] T 4d282d79e1664f7ea8912e0c5166eec3 P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:55.850064 8703 raft_consensus.cc:482] T 4d282d79e1664f7ea8912e0c5166eec3 P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:55.850117 8703 raft_consensus.cc:3037] T 4d282d79e1664f7ea8912e0c5166eec3 P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:55.874963 8709 raft_consensus.cc:504] T 2880045de45743bdb5358a17d93b2245 P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:08:55.874965 8703 raft_consensus.cc:504] T 4d282d79e1664f7ea8912e0c5166eec3 P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } }
I20240501 14:08:55.875525 8703 leader_election.cc:288] T 4d282d79e1664f7ea8912e0c5166eec3 P 47eabc135e9f4237b036097085e729d8 [CANDIDATE]: Term 1 election: Requested vote from peers 9ae5a87ca07b40a788a42734b24f66c6 (127.7.39.131:34697), ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083)
I20240501 14:08:55.875542 8709 leader_election.cc:288] T 2880045de45743bdb5358a17d93b2245 P 9ae5a87ca07b40a788a42734b24f66c6 [CANDIDATE]: Term 1 election: Requested vote from peers ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083), 9e44efa72e7d4421b284ae74d1cb9634 (127.7.39.130:46433)
I20240501 14:08:55.876269 7511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d282d79e1664f7ea8912e0c5166eec3" candidate_uuid: "47eabc135e9f4237b036097085e729d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae52f888f5094416a24cb3dffbff03c4"
I20240501 14:08:55.876355 8700 raft_consensus.cc:504] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } }
I20240501 14:08:55.876413 7511 raft_consensus.cc:3037] T 4d282d79e1664f7ea8912e0c5166eec3 P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:55.876423 7510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2880045de45743bdb5358a17d93b2245" candidate_uuid: "9ae5a87ca07b40a788a42734b24f66c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae52f888f5094416a24cb3dffbff03c4"
I20240501 14:08:55.876696 7510 raft_consensus.cc:3037] T 2880045de45743bdb5358a17d93b2245 P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:55.876730 8700 leader_election.cc:288] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6 [CANDIDATE]: Term 1 election: Requested vote from peers 9e44efa72e7d4421b284ae74d1cb9634 (127.7.39.130:46433), 47eabc135e9f4237b036097085e729d8 (127.7.39.132:42369)
I20240501 14:08:55.877204 7903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6841e1b32b86454f9e40b64c141a2a6b" candidate_uuid: "9ae5a87ca07b40a788a42734b24f66c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47eabc135e9f4237b036097085e729d8"
I20240501 14:08:55.877341 7903 raft_consensus.cc:3037] T 6841e1b32b86454f9e40b64c141a2a6b P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:55.877542 7511 raft_consensus.cc:2454] T 4d282d79e1664f7ea8912e0c5166eec3 P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47eabc135e9f4237b036097085e729d8 in term 1.
I20240501 14:08:55.877777 7772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d282d79e1664f7ea8912e0c5166eec3" candidate_uuid: "47eabc135e9f4237b036097085e729d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ae5a87ca07b40a788a42734b24f66c6"
I20240501 14:08:55.877894 7772 raft_consensus.cc:3037] T 4d282d79e1664f7ea8912e0c5166eec3 P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:55.877888 7839 leader_election.cc:302] T 4d282d79e1664f7ea8912e0c5166eec3 P 47eabc135e9f4237b036097085e729d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47eabc135e9f4237b036097085e729d8, ae52f888f5094416a24cb3dffbff03c4; no voters:
I20240501 14:08:55.878093 8703 raft_consensus.cc:2781] T 4d282d79e1664f7ea8912e0c5166eec3 P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:55.878180 8703 raft_consensus.cc:686] T 4d282d79e1664f7ea8912e0c5166eec3 P 47eabc135e9f4237b036097085e729d8 [term 1 LEADER]: Becoming Leader. State: Replica: 47eabc135e9f4237b036097085e729d8, State: Running, Role: LEADER
I20240501 14:08:55.878367 8703 consensus_queue.cc:237] T 4d282d79e1664f7ea8912e0c5166eec3 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } }
I20240501 14:08:55.878615 7903 raft_consensus.cc:2454] T 6841e1b32b86454f9e40b64c141a2a6b P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ae5a87ca07b40a788a42734b24f66c6 in term 1.
I20240501 14:08:55.879200 7707 leader_election.cc:302] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47eabc135e9f4237b036097085e729d8, 9ae5a87ca07b40a788a42734b24f66c6; no voters:
I20240501 14:08:55.879367 8700 raft_consensus.cc:2781] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:55.879465 8700 raft_consensus.cc:686] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 LEADER]: Becoming Leader. State: Replica: 9ae5a87ca07b40a788a42734b24f66c6, State: Running, Role: LEADER
I20240501 14:08:55.879796 8700 consensus_queue.cc:237] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } }
I20240501 14:08:55.880016 7641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6841e1b32b86454f9e40b64c141a2a6b" candidate_uuid: "9ae5a87ca07b40a788a42734b24f66c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e44efa72e7d4421b284ae74d1cb9634"
I20240501 14:08:55.880159 7641 raft_consensus.cc:3037] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:55.880285 7642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2880045de45743bdb5358a17d93b2245" candidate_uuid: "9ae5a87ca07b40a788a42734b24f66c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e44efa72e7d4421b284ae74d1cb9634"
I20240501 14:08:55.880394 7642 raft_consensus.cc:3037] T 2880045de45743bdb5358a17d93b2245 P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:55.880712 7367 catalog_manager.cc:5484] T 4d282d79e1664f7ea8912e0c5166eec3 P 47eabc135e9f4237b036097085e729d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 47eabc135e9f4237b036097085e729d8 (127.7.39.132). New cstate: current_term: 1 leader_uuid: "47eabc135e9f4237b036097085e729d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } health_report { overall_health: HEALTHY } } }
I20240501 14:08:55.882584 7364 catalog_manager.cc:5484] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ae5a87ca07b40a788a42734b24f66c6 (127.7.39.131). New cstate: current_term: 1 leader_uuid: "9ae5a87ca07b40a788a42734b24f66c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } health_report { overall_health: UNKNOWN } } }
I20240501 14:08:55.973729 7510 raft_consensus.cc:2454] T 2880045de45743bdb5358a17d93b2245 P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ae5a87ca07b40a788a42734b24f66c6 in term 1.
I20240501 14:08:55.973729 7641 raft_consensus.cc:2454] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ae5a87ca07b40a788a42734b24f66c6 in term 1.
I20240501 14:08:55.973762 7772 raft_consensus.cc:2454] T 4d282d79e1664f7ea8912e0c5166eec3 P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47eabc135e9f4237b036097085e729d8 in term 1.
I20240501 14:08:55.974277 7642 raft_consensus.cc:2454] T 2880045de45743bdb5358a17d93b2245 P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ae5a87ca07b40a788a42734b24f66c6 in term 1.
I20240501 14:08:55.974500 7708 leader_election.cc:302] T 2880045de45743bdb5358a17d93b2245 P 9ae5a87ca07b40a788a42734b24f66c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ae5a87ca07b40a788a42734b24f66c6, ae52f888f5094416a24cb3dffbff03c4; no voters:
I20240501 14:08:55.974723 8700 raft_consensus.cc:2781] T 2880045de45743bdb5358a17d93b2245 P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:55.974805 8700 raft_consensus.cc:686] T 2880045de45743bdb5358a17d93b2245 P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 LEADER]: Becoming Leader. State: Replica: 9ae5a87ca07b40a788a42734b24f66c6, State: Running, Role: LEADER
I20240501 14:08:55.974941 8700 consensus_queue.cc:237] T 2880045de45743bdb5358a17d93b2245 P 9ae5a87ca07b40a788a42734b24f66c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:08:55.976846 7364 catalog_manager.cc:5484] T 2880045de45743bdb5358a17d93b2245 P 9ae5a87ca07b40a788a42734b24f66c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ae5a87ca07b40a788a42734b24f66c6 (127.7.39.131). New cstate: current_term: 1 leader_uuid: "9ae5a87ca07b40a788a42734b24f66c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } health_report { overall_health: UNKNOWN } } }
W20240501 14:08:56.125423 8736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:56.125680 8736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:56.128749 8736 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240501 14:08:56.226398 7903 raft_consensus.cc:1261] T 6841e1b32b86454f9e40b64c141a2a6b P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Refusing update from remote peer 9ae5a87ca07b40a788a42734b24f66c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:56.227283 7642 raft_consensus.cc:1261] T 2880045de45743bdb5358a17d93b2245 P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 FOLLOWER]: Refusing update from remote peer 9ae5a87ca07b40a788a42734b24f66c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:56.227958 8709 consensus_queue.cc:1035] T 2880045de45743bdb5358a17d93b2245 P 9ae5a87ca07b40a788a42734b24f66c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:56.228310 7510 raft_consensus.cc:1261] T 2880045de45743bdb5358a17d93b2245 P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Refusing update from remote peer 9ae5a87ca07b40a788a42734b24f66c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:56.229315 8700 consensus_queue.cc:1035] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:56.229983 7772 raft_consensus.cc:1261] T 4d282d79e1664f7ea8912e0c5166eec3 P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:56.230487 7510 raft_consensus.cc:1261] T 4d282d79e1664f7ea8912e0c5166eec3 P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:56.230537 8709 consensus_queue.cc:1035] T 2880045de45743bdb5358a17d93b2245 P 9ae5a87ca07b40a788a42734b24f66c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:56.230605 7641 raft_consensus.cc:1261] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 FOLLOWER]: Refusing update from remote peer 9ae5a87ca07b40a788a42734b24f66c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:56.230963 8703 consensus_queue.cc:1035] T 4d282d79e1664f7ea8912e0c5166eec3 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:56.232679 8567 consensus_queue.cc:1035] T 4d282d79e1664f7ea8912e0c5166eec3 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:56.235350 8709 consensus_queue.cc:1035] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 127.126 ms
time per row: 0.0635632 ms
I20240501 14:08:56.301065 7365 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32870:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:08:56.306591 7882 tablet_service.cc:1467] Processing CreateTablet for tablet df856440b07b418394477cf368b765c1 (DEFAULT_TABLE table=rebalance_test_table_4 [id=9b4ecb4938fd4bc185e20ce3ec5254f5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:56.306592 7491 tablet_service.cc:1467] Processing CreateTablet for tablet df856440b07b418394477cf368b765c1 (DEFAULT_TABLE table=rebalance_test_table_4 [id=9b4ecb4938fd4bc185e20ce3ec5254f5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:56.306929 7491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df856440b07b418394477cf368b765c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:56.306928 7882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df856440b07b418394477cf368b765c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:56.307500 7622 tablet_service.cc:1467] Processing CreateTablet for tablet df856440b07b418394477cf368b765c1 (DEFAULT_TABLE table=rebalance_test_table_4 [id=9b4ecb4938fd4bc185e20ce3ec5254f5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:56.307768 7622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df856440b07b418394477cf368b765c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:56.309247 8794 tablet_bootstrap.cc:492] T df856440b07b418394477cf368b765c1 P ae52f888f5094416a24cb3dffbff03c4: Bootstrap starting.
I20240501 14:08:56.309576 7620 tablet_service.cc:1467] Processing CreateTablet for tablet dcf673c75a8b48da95806e1c43309dda (DEFAULT_TABLE table=rebalance_test_table_4 [id=9b4ecb4938fd4bc185e20ce3ec5254f5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:56.309684 7753 tablet_service.cc:1467] Processing CreateTablet for tablet dcf673c75a8b48da95806e1c43309dda (DEFAULT_TABLE table=rebalance_test_table_4 [id=9b4ecb4938fd4bc185e20ce3ec5254f5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:56.309809 7620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcf673c75a8b48da95806e1c43309dda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:56.309935 7753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcf673c75a8b48da95806e1c43309dda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:56.310140 8794 tablet_bootstrap.cc:654] T df856440b07b418394477cf368b765c1 P ae52f888f5094416a24cb3dffbff03c4: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:56.310761 8793 tablet_bootstrap.cc:492] T df856440b07b418394477cf368b765c1 P 47eabc135e9f4237b036097085e729d8: Bootstrap starting.
I20240501 14:08:56.311551 8794 tablet_bootstrap.cc:492] T df856440b07b418394477cf368b765c1 P ae52f888f5094416a24cb3dffbff03c4: No bootstrap required, opened a new log
I20240501 14:08:56.311640 8794 ts_tablet_manager.cc:1367] T df856440b07b418394477cf368b765c1 P ae52f888f5094416a24cb3dffbff03c4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:56.311743 8793 tablet_bootstrap.cc:654] T df856440b07b418394477cf368b765c1 P 47eabc135e9f4237b036097085e729d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:56.312440 8795 tablet_bootstrap.cc:492] T df856440b07b418394477cf368b765c1 P 9e44efa72e7d4421b284ae74d1cb9634: Bootstrap starting.
I20240501 14:08:56.312402 7752 tablet_service.cc:1467] Processing CreateTablet for tablet c747bd6504c540cebded0019d82495b4 (DEFAULT_TABLE table=rebalance_test_table_4 [id=9b4ecb4938fd4bc185e20ce3ec5254f5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:56.312647 7752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c747bd6504c540cebded0019d82495b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:56.312355 8794 raft_consensus.cc:348] T df856440b07b418394477cf368b765c1 P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:08:56.312938 7489 tablet_service.cc:1467] Processing CreateTablet for tablet c747bd6504c540cebded0019d82495b4 (DEFAULT_TABLE table=rebalance_test_table_4 [id=9b4ecb4938fd4bc185e20ce3ec5254f5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:56.313166 7489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c747bd6504c540cebded0019d82495b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:56.313179 7882 tablet_service.cc:1467] Processing CreateTablet for tablet c747bd6504c540cebded0019d82495b4 (DEFAULT_TABLE table=rebalance_test_table_4 [id=9b4ecb4938fd4bc185e20ce3ec5254f5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:56.313414 7882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c747bd6504c540cebded0019d82495b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:56.313550 8797 tablet_bootstrap.cc:492] T dcf673c75a8b48da95806e1c43309dda P 9ae5a87ca07b40a788a42734b24f66c6: Bootstrap starting.
I20240501 14:08:56.314417 8797 tablet_bootstrap.cc:654] T dcf673c75a8b48da95806e1c43309dda P 9ae5a87ca07b40a788a42734b24f66c6: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:56.315809 8794 raft_consensus.cc:374] T df856440b07b418394477cf368b765c1 P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:56.315881 8794 raft_consensus.cc:729] T df856440b07b418394477cf368b765c1 P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae52f888f5094416a24cb3dffbff03c4, State: Initialized, Role: FOLLOWER
I20240501 14:08:56.315914 8795 tablet_bootstrap.cc:654] T df856440b07b418394477cf368b765c1 P 9e44efa72e7d4421b284ae74d1cb9634: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:56.315994 8793 tablet_bootstrap.cc:492] T df856440b07b418394477cf368b765c1 P 47eabc135e9f4237b036097085e729d8: No bootstrap required, opened a new log
I20240501 14:08:56.316072 8793 ts_tablet_manager.cc:1367] T df856440b07b418394477cf368b765c1 P 47eabc135e9f4237b036097085e729d8: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:08:56.316048 8794 consensus_queue.cc:260] T df856440b07b418394477cf368b765c1 P ae52f888f5094416a24cb3dffbff03c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:08:56.316526 8793 raft_consensus.cc:348] T df856440b07b418394477cf368b765c1 P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:08:56.316630 8793 raft_consensus.cc:374] T df856440b07b418394477cf368b765c1 P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:56.316665 8793 raft_consensus.cc:729] T df856440b07b418394477cf368b765c1 P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47eabc135e9f4237b036097085e729d8, State: Initialized, Role: FOLLOWER
I20240501 14:08:56.316780 8793 consensus_queue.cc:260] T df856440b07b418394477cf368b765c1 P 47eabc135e9f4237b036097085e729d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:08:56.317073 8793 ts_tablet_manager.cc:1398] T df856440b07b418394477cf368b765c1 P 47eabc135e9f4237b036097085e729d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:56.317248 8793 tablet_bootstrap.cc:492] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8: Bootstrap starting.
I20240501 14:08:56.317595 8794 ts_tablet_manager.cc:1398] T df856440b07b418394477cf368b765c1 P ae52f888f5094416a24cb3dffbff03c4: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:08:56.317775 8794 tablet_bootstrap.cc:492] T c747bd6504c540cebded0019d82495b4 P ae52f888f5094416a24cb3dffbff03c4: Bootstrap starting.
I20240501 14:08:56.318126 8793 tablet_bootstrap.cc:654] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:56.318739 8794 tablet_bootstrap.cc:654] T c747bd6504c540cebded0019d82495b4 P ae52f888f5094416a24cb3dffbff03c4: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:56.319828 8793 tablet_bootstrap.cc:492] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8: No bootstrap required, opened a new log
I20240501 14:08:56.319852 7491 tablet_service.cc:1467] Processing CreateTablet for tablet dcf673c75a8b48da95806e1c43309dda (DEFAULT_TABLE table=rebalance_test_table_4 [id=9b4ecb4938fd4bc185e20ce3ec5254f5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:56.319916 8793 ts_tablet_manager.cc:1367] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:08:56.320109 7491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcf673c75a8b48da95806e1c43309dda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:56.320159 8794 tablet_bootstrap.cc:492] T c747bd6504c540cebded0019d82495b4 P ae52f888f5094416a24cb3dffbff03c4: No bootstrap required, opened a new log
I20240501 14:08:56.320237 8794 ts_tablet_manager.cc:1367] T c747bd6504c540cebded0019d82495b4 P ae52f888f5094416a24cb3dffbff03c4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:56.320348 8793 raft_consensus.cc:348] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:08:56.320449 8793 raft_consensus.cc:374] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:56.320479 8793 raft_consensus.cc:729] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47eabc135e9f4237b036097085e729d8, State: Initialized, Role: FOLLOWER
I20240501 14:08:56.320632 8793 consensus_queue.cc:260] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:08:56.320623 8794 raft_consensus.cc:348] T c747bd6504c540cebded0019d82495b4 P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:08:56.320739 8794 raft_consensus.cc:374] T c747bd6504c540cebded0019d82495b4 P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:56.320775 8794 raft_consensus.cc:729] T c747bd6504c540cebded0019d82495b4 P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae52f888f5094416a24cb3dffbff03c4, State: Initialized, Role: FOLLOWER
I20240501 14:08:56.320917 8793 ts_tablet_manager.cc:1398] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:56.321542 8794 consensus_queue.cc:260] T c747bd6504c540cebded0019d82495b4 P ae52f888f5094416a24cb3dffbff03c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:08:56.321992 8794 ts_tablet_manager.cc:1398] T c747bd6504c540cebded0019d82495b4 P ae52f888f5094416a24cb3dffbff03c4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:56.324610 8797 tablet_bootstrap.cc:492] T dcf673c75a8b48da95806e1c43309dda P 9ae5a87ca07b40a788a42734b24f66c6: No bootstrap required, opened a new log
I20240501 14:08:56.324723 8797 ts_tablet_manager.cc:1367] T dcf673c75a8b48da95806e1c43309dda P 9ae5a87ca07b40a788a42734b24f66c6: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20240501 14:08:56.325009 8794 tablet_bootstrap.cc:492] T dcf673c75a8b48da95806e1c43309dda P ae52f888f5094416a24cb3dffbff03c4: Bootstrap starting.
I20240501 14:08:56.325206 8797 raft_consensus.cc:348] T dcf673c75a8b48da95806e1c43309dda P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:08:56.325323 8797 raft_consensus.cc:374] T dcf673c75a8b48da95806e1c43309dda P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:56.326146 8794 tablet_bootstrap.cc:654] T dcf673c75a8b48da95806e1c43309dda P ae52f888f5094416a24cb3dffbff03c4: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:56.326228 8797 raft_consensus.cc:729] T dcf673c75a8b48da95806e1c43309dda P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ae5a87ca07b40a788a42734b24f66c6, State: Initialized, Role: FOLLOWER
I20240501 14:08:56.326566 8797 consensus_queue.cc:260] T dcf673c75a8b48da95806e1c43309dda P 9ae5a87ca07b40a788a42734b24f66c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:08:56.327026 8797 ts_tablet_manager.cc:1398] T dcf673c75a8b48da95806e1c43309dda P 9ae5a87ca07b40a788a42734b24f66c6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:56.327240 8797 tablet_bootstrap.cc:492] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6: Bootstrap starting.
I20240501 14:08:56.327981 8794 tablet_bootstrap.cc:492] T dcf673c75a8b48da95806e1c43309dda P ae52f888f5094416a24cb3dffbff03c4: No bootstrap required, opened a new log
I20240501 14:08:56.328092 8794 ts_tablet_manager.cc:1367] T dcf673c75a8b48da95806e1c43309dda P ae52f888f5094416a24cb3dffbff03c4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:08:56.328675 8794 raft_consensus.cc:348] T dcf673c75a8b48da95806e1c43309dda P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:08:56.328780 8794 raft_consensus.cc:374] T dcf673c75a8b48da95806e1c43309dda P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:56.329007 8794 raft_consensus.cc:729] T dcf673c75a8b48da95806e1c43309dda P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae52f888f5094416a24cb3dffbff03c4, State: Initialized, Role: FOLLOWER
I20240501 14:08:56.329138 8794 consensus_queue.cc:260] T dcf673c75a8b48da95806e1c43309dda P ae52f888f5094416a24cb3dffbff03c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:08:56.328910 8797 tablet_bootstrap.cc:654] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:56.329834 8794 ts_tablet_manager.cc:1398] T dcf673c75a8b48da95806e1c43309dda P ae52f888f5094416a24cb3dffbff03c4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:56.330924 8797 tablet_bootstrap.cc:492] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6: No bootstrap required, opened a new log
I20240501 14:08:56.330997 8797 ts_tablet_manager.cc:1367] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:08:56.331339 8795 tablet_bootstrap.cc:492] T df856440b07b418394477cf368b765c1 P 9e44efa72e7d4421b284ae74d1cb9634: No bootstrap required, opened a new log
I20240501 14:08:56.331444 8795 ts_tablet_manager.cc:1367] T df856440b07b418394477cf368b765c1 P 9e44efa72e7d4421b284ae74d1cb9634: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240501 14:08:56.331424 8797 raft_consensus.cc:348] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:08:56.331526 8797 raft_consensus.cc:374] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:56.331555 8797 raft_consensus.cc:729] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ae5a87ca07b40a788a42734b24f66c6, State: Initialized, Role: FOLLOWER
I20240501 14:08:56.331660 8797 consensus_queue.cc:260] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:08:56.331885 8797 ts_tablet_manager.cc:1398] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:08:56.331926 8795 raft_consensus.cc:348] T df856440b07b418394477cf368b765c1 P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:08:56.332038 8795 raft_consensus.cc:374] T df856440b07b418394477cf368b765c1 P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:56.332082 8795 raft_consensus.cc:729] T df856440b07b418394477cf368b765c1 P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e44efa72e7d4421b284ae74d1cb9634, State: Initialized, Role: FOLLOWER
I20240501 14:08:56.332208 8795 consensus_queue.cc:260] T df856440b07b418394477cf368b765c1 P 9e44efa72e7d4421b284ae74d1cb9634 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:08:56.332506 8795 ts_tablet_manager.cc:1398] T df856440b07b418394477cf368b765c1 P 9e44efa72e7d4421b284ae74d1cb9634: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:08:56.332710 8795 tablet_bootstrap.cc:492] T dcf673c75a8b48da95806e1c43309dda P 9e44efa72e7d4421b284ae74d1cb9634: Bootstrap starting.
I20240501 14:08:56.333714 8795 tablet_bootstrap.cc:654] T dcf673c75a8b48da95806e1c43309dda P 9e44efa72e7d4421b284ae74d1cb9634: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:56.334930 8795 tablet_bootstrap.cc:492] T dcf673c75a8b48da95806e1c43309dda P 9e44efa72e7d4421b284ae74d1cb9634: No bootstrap required, opened a new log
I20240501 14:08:56.335023 8795 ts_tablet_manager.cc:1367] T dcf673c75a8b48da95806e1c43309dda P 9e44efa72e7d4421b284ae74d1cb9634: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:56.335523 8795 raft_consensus.cc:348] T dcf673c75a8b48da95806e1c43309dda P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:08:56.335626 8795 raft_consensus.cc:374] T dcf673c75a8b48da95806e1c43309dda P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:56.335660 8795 raft_consensus.cc:729] T dcf673c75a8b48da95806e1c43309dda P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e44efa72e7d4421b284ae74d1cb9634, State: Initialized, Role: FOLLOWER
I20240501 14:08:56.335809 8795 consensus_queue.cc:260] T dcf673c75a8b48da95806e1c43309dda P 9e44efa72e7d4421b284ae74d1cb9634 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:08:56.336107 8795 ts_tablet_manager.cc:1398] T dcf673c75a8b48da95806e1c43309dda P 9e44efa72e7d4421b284ae74d1cb9634: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:08:56.358747 8654 raft_consensus.cc:482] T df856440b07b418394477cf368b765c1 P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:56.358876 8654 raft_consensus.cc:504] T df856440b07b418394477cf368b765c1 P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:08:56.359329 8654 leader_election.cc:288] T df856440b07b418394477cf368b765c1 P ae52f888f5094416a24cb3dffbff03c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47eabc135e9f4237b036097085e729d8 (127.7.39.132:42369), 9e44efa72e7d4421b284ae74d1cb9634 (127.7.39.130:46433)
I20240501 14:08:56.359723 7903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df856440b07b418394477cf368b765c1" candidate_uuid: "ae52f888f5094416a24cb3dffbff03c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47eabc135e9f4237b036097085e729d8" is_pre_election: true
I20240501 14:08:56.359747 7642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df856440b07b418394477cf368b765c1" candidate_uuid: "ae52f888f5094416a24cb3dffbff03c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e44efa72e7d4421b284ae74d1cb9634" is_pre_election: true
I20240501 14:08:56.359887 7903 raft_consensus.cc:2454] T df856440b07b418394477cf368b765c1 P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae52f888f5094416a24cb3dffbff03c4 in term 0.
I20240501 14:08:56.359887 7642 raft_consensus.cc:2454] T df856440b07b418394477cf368b765c1 P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae52f888f5094416a24cb3dffbff03c4 in term 0.
I20240501 14:08:56.360198 7446 leader_election.cc:302] T df856440b07b418394477cf368b765c1 P ae52f888f5094416a24cb3dffbff03c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e44efa72e7d4421b284ae74d1cb9634, ae52f888f5094416a24cb3dffbff03c4; no voters:
I20240501 14:08:56.360374 8654 raft_consensus.cc:2781] T df856440b07b418394477cf368b765c1 P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:56.360452 8654 raft_consensus.cc:482] T df856440b07b418394477cf368b765c1 P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:56.360494 8654 raft_consensus.cc:3037] T df856440b07b418394477cf368b765c1 P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:56.375829 8654 raft_consensus.cc:504] T df856440b07b418394477cf368b765c1 P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:08:56.376430 8654 leader_election.cc:288] T df856440b07b418394477cf368b765c1 P ae52f888f5094416a24cb3dffbff03c4 [CANDIDATE]: Term 1 election: Requested vote from peers 47eabc135e9f4237b036097085e729d8 (127.7.39.132:42369), 9e44efa72e7d4421b284ae74d1cb9634 (127.7.39.130:46433)
I20240501 14:08:56.376864 7903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df856440b07b418394477cf368b765c1" candidate_uuid: "ae52f888f5094416a24cb3dffbff03c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47eabc135e9f4237b036097085e729d8"
I20240501 14:08:56.377032 7903 raft_consensus.cc:3037] T df856440b07b418394477cf368b765c1 P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:56.377030 7642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df856440b07b418394477cf368b765c1" candidate_uuid: "ae52f888f5094416a24cb3dffbff03c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e44efa72e7d4421b284ae74d1cb9634"
I20240501 14:08:56.377153 7642 raft_consensus.cc:3037] T df856440b07b418394477cf368b765c1 P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:56.378090 7642 raft_consensus.cc:2454] T df856440b07b418394477cf368b765c1 P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae52f888f5094416a24cb3dffbff03c4 in term 1.
I20240501 14:08:56.378091 7903 raft_consensus.cc:2454] T df856440b07b418394477cf368b765c1 P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae52f888f5094416a24cb3dffbff03c4 in term 1.
I20240501 14:08:56.378490 7445 leader_election.cc:302] T df856440b07b418394477cf368b765c1 P ae52f888f5094416a24cb3dffbff03c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47eabc135e9f4237b036097085e729d8, ae52f888f5094416a24cb3dffbff03c4; no voters:
I20240501 14:08:56.378695 8654 raft_consensus.cc:2781] T df856440b07b418394477cf368b765c1 P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:56.379112 8654 raft_consensus.cc:686] T df856440b07b418394477cf368b765c1 P ae52f888f5094416a24cb3dffbff03c4 [term 1 LEADER]: Becoming Leader. State: Replica: ae52f888f5094416a24cb3dffbff03c4, State: Running, Role: LEADER
I20240501 14:08:56.379472 8654 consensus_queue.cc:237] T df856440b07b418394477cf368b765c1 P ae52f888f5094416a24cb3dffbff03c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:08:56.382524 7360 catalog_manager.cc:5484] T df856440b07b418394477cf368b765c1 P ae52f888f5094416a24cb3dffbff03c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129). New cstate: current_term: 1 leader_uuid: "ae52f888f5094416a24cb3dffbff03c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } health_report { overall_health: UNKNOWN } } }
I20240501 14:08:56.616087 8700 raft_consensus.cc:482] T dcf673c75a8b48da95806e1c43309dda P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:56.616240 8700 raft_consensus.cc:504] T dcf673c75a8b48da95806e1c43309dda P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:08:56.616627 8700 leader_election.cc:288] T dcf673c75a8b48da95806e1c43309dda P 9ae5a87ca07b40a788a42734b24f66c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083), 9e44efa72e7d4421b284ae74d1cb9634 (127.7.39.130:46433)
I20240501 14:08:56.617166 7510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcf673c75a8b48da95806e1c43309dda" candidate_uuid: "9ae5a87ca07b40a788a42734b24f66c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae52f888f5094416a24cb3dffbff03c4" is_pre_election: true
I20240501 14:08:56.617273 7642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcf673c75a8b48da95806e1c43309dda" candidate_uuid: "9ae5a87ca07b40a788a42734b24f66c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e44efa72e7d4421b284ae74d1cb9634" is_pre_election: true
I20240501 14:08:56.617353 7510 raft_consensus.cc:2454] T dcf673c75a8b48da95806e1c43309dda P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ae5a87ca07b40a788a42734b24f66c6 in term 0.
I20240501 14:08:56.617421 7642 raft_consensus.cc:2454] T dcf673c75a8b48da95806e1c43309dda P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ae5a87ca07b40a788a42734b24f66c6 in term 0.
I20240501 14:08:56.617745 7708 leader_election.cc:302] T dcf673c75a8b48da95806e1c43309dda P 9ae5a87ca07b40a788a42734b24f66c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ae5a87ca07b40a788a42734b24f66c6, ae52f888f5094416a24cb3dffbff03c4; no voters:
I20240501 14:08:56.617977 8700 raft_consensus.cc:2781] T dcf673c75a8b48da95806e1c43309dda P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:56.618036 8700 raft_consensus.cc:482] T dcf673c75a8b48da95806e1c43309dda P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:56.618064 8700 raft_consensus.cc:3037] T dcf673c75a8b48da95806e1c43309dda P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:56.650246 8703 raft_consensus.cc:482] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:56.650411 8703 raft_consensus.cc:504] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:08:56.650885 8703 leader_election.cc:288] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083), 9ae5a87ca07b40a788a42734b24f66c6 (127.7.39.131:34697)
I20240501 14:08:56.651448 7772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c747bd6504c540cebded0019d82495b4" candidate_uuid: "47eabc135e9f4237b036097085e729d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ae5a87ca07b40a788a42734b24f66c6" is_pre_election: true
I20240501 14:08:56.651448 7510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c747bd6504c540cebded0019d82495b4" candidate_uuid: "47eabc135e9f4237b036097085e729d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae52f888f5094416a24cb3dffbff03c4" is_pre_election: true
I20240501 14:08:56.651624 7772 raft_consensus.cc:2454] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47eabc135e9f4237b036097085e729d8 in term 0.
I20240501 14:08:56.651624 7510 raft_consensus.cc:2454] T c747bd6504c540cebded0019d82495b4 P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47eabc135e9f4237b036097085e729d8 in term 0.
I20240501 14:08:56.651963 7839 leader_election.cc:302] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47eabc135e9f4237b036097085e729d8, 9ae5a87ca07b40a788a42734b24f66c6; no voters:
I20240501 14:08:56.652143 8703 raft_consensus.cc:2781] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:56.652215 8703 raft_consensus.cc:482] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:56.652271 8703 raft_consensus.cc:3037] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:56.678457 8700 raft_consensus.cc:504] T dcf673c75a8b48da95806e1c43309dda P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:08:56.678455 8703 raft_consensus.cc:504] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:08:56.678977 8700 leader_election.cc:288] T dcf673c75a8b48da95806e1c43309dda P 9ae5a87ca07b40a788a42734b24f66c6 [CANDIDATE]: Term 1 election: Requested vote from peers ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083), 9e44efa72e7d4421b284ae74d1cb9634 (127.7.39.130:46433)
I20240501 14:08:56.679116 8703 leader_election.cc:288] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [CANDIDATE]: Term 1 election: Requested vote from peers ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083), 9ae5a87ca07b40a788a42734b24f66c6 (127.7.39.131:34697)
I20240501 14:08:56.679724 7772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c747bd6504c540cebded0019d82495b4" candidate_uuid: "47eabc135e9f4237b036097085e729d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ae5a87ca07b40a788a42734b24f66c6"
I20240501 14:08:56.679713 7510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c747bd6504c540cebded0019d82495b4" candidate_uuid: "47eabc135e9f4237b036097085e729d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae52f888f5094416a24cb3dffbff03c4"
I20240501 14:08:56.679854 7772 raft_consensus.cc:3037] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:56.679870 7510 raft_consensus.cc:3037] T c747bd6504c540cebded0019d82495b4 P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:56.679713 7511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcf673c75a8b48da95806e1c43309dda" candidate_uuid: "9ae5a87ca07b40a788a42734b24f66c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae52f888f5094416a24cb3dffbff03c4"
I20240501 14:08:56.680454 7511 raft_consensus.cc:3037] T dcf673c75a8b48da95806e1c43309dda P ae52f888f5094416a24cb3dffbff03c4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:56.680902 7772 raft_consensus.cc:2454] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47eabc135e9f4237b036097085e729d8 in term 1.
I20240501 14:08:56.681481 7511 raft_consensus.cc:2454] T dcf673c75a8b48da95806e1c43309dda P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ae5a87ca07b40a788a42734b24f66c6 in term 1.
I20240501 14:08:56.681592 7510 raft_consensus.cc:2454] T c747bd6504c540cebded0019d82495b4 P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47eabc135e9f4237b036097085e729d8 in term 1.
I20240501 14:08:56.681602 7839 leader_election.cc:302] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47eabc135e9f4237b036097085e729d8, 9ae5a87ca07b40a788a42734b24f66c6; no voters:
I20240501 14:08:56.681823 7708 leader_election.cc:302] T dcf673c75a8b48da95806e1c43309dda P 9ae5a87ca07b40a788a42734b24f66c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ae5a87ca07b40a788a42734b24f66c6, ae52f888f5094416a24cb3dffbff03c4; no voters:
I20240501 14:08:56.682005 8700 raft_consensus.cc:2781] T dcf673c75a8b48da95806e1c43309dda P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:56.681998 7642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcf673c75a8b48da95806e1c43309dda" candidate_uuid: "9ae5a87ca07b40a788a42734b24f66c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e44efa72e7d4421b284ae74d1cb9634"
I20240501 14:08:56.682073 8700 raft_consensus.cc:686] T dcf673c75a8b48da95806e1c43309dda P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 LEADER]: Becoming Leader. State: Replica: 9ae5a87ca07b40a788a42734b24f66c6, State: Running, Role: LEADER
I20240501 14:08:56.682055 8703 raft_consensus.cc:2781] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:56.682127 7642 raft_consensus.cc:3037] T dcf673c75a8b48da95806e1c43309dda P 9e44efa72e7d4421b284ae74d1cb9634 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:56.682152 8703 raft_consensus.cc:686] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [term 1 LEADER]: Becoming Leader. State: Replica: 47eabc135e9f4237b036097085e729d8, State: Running, Role: LEADER
I20240501 14:08:56.682214 8700 consensus_queue.cc:237] T dcf673c75a8b48da95806e1c43309dda P 9ae5a87ca07b40a788a42734b24f66c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:08:56.682319 8703 consensus_queue.cc:237] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:08:56.684443 7365 catalog_manager.cc:5484] T dcf673c75a8b48da95806e1c43309dda P 9ae5a87ca07b40a788a42734b24f66c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ae5a87ca07b40a788a42734b24f66c6 (127.7.39.131). New cstate: current_term: 1 leader_uuid: "9ae5a87ca07b40a788a42734b24f66c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } health_report { overall_health: UNKNOWN } } }
I20240501 14:08:56.684443 7360 catalog_manager.cc:5484] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 47eabc135e9f4237b036097085e729d8 (127.7.39.132). New cstate: current_term: 1 leader_uuid: "47eabc135e9f4237b036097085e729d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } health_report { overall_health: UNKNOWN } } }
I20240501 14:08:56.774677 7642 raft_consensus.cc:2454] T dcf673c75a8b48da95806e1c43309dda P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ae5a87ca07b40a788a42734b24f66c6 in term 1.
W20240501 14:08:56.889043 8811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:56.889302 8811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:56.892493 8811 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240501 14:08:56.960168 8654 consensus_queue.cc:1035] T df856440b07b418394477cf368b765c1 P ae52f888f5094416a24cb3dffbff03c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:56.967320 8836 consensus_queue.cc:1035] T df856440b07b418394477cf368b765c1 P ae52f888f5094416a24cb3dffbff03c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:56.979986 7510 raft_consensus.cc:1261] T c747bd6504c540cebded0019d82495b4 P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:56.980484 8567 consensus_queue.cc:1035] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:56.980810 7772 raft_consensus.cc:1261] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:56.981333 8567 consensus_queue.cc:1035] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:57.000873 7642 raft_consensus.cc:1261] T dcf673c75a8b48da95806e1c43309dda P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 FOLLOWER]: Refusing update from remote peer 9ae5a87ca07b40a788a42734b24f66c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:57.001385 8700 consensus_queue.cc:1035] T dcf673c75a8b48da95806e1c43309dda P 9ae5a87ca07b40a788a42734b24f66c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:57.002323 7510 raft_consensus.cc:1261] T dcf673c75a8b48da95806e1c43309dda P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Refusing update from remote peer 9ae5a87ca07b40a788a42734b24f66c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:57.002738 8709 consensus_queue.cc:1035] T dcf673c75a8b48da95806e1c43309dda P 9ae5a87ca07b40a788a42734b24f66c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 115.201 ms
time per row: 0.0576004 ms
I20240501 14:08:57.052898 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.133:38805
--local_ip_for_outbound_sockets=127.7.39.133
--tserver_master_addrs=127.7.39.190:44225
--webserver_port=33043
--webserver_interface=127.7.39.133
--builtin_ntp_servers=127.7.39.148:46693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240501 14:08:57.173432 8871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:57.173691 8871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:57.173770 8871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:57.177310 8871 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:08:57.177484 8871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:57.177706 8871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.133
I20240501 14:08:57.184789 8871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:46693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.133:38805
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.7.39.133
--webserver_port=33043
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.7.39.190:44225
--never_fsync=true
--heap_profile_path=/tmp/kudu.8871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.133
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:57.186014 8871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:57.187820 8871 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:57.199252 8877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:57.200384 8876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:57.201627 8879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:57.202915 8871 server_base.cc:1034] running on GCE node
I20240501 14:08:57.203380 8871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:57.203986 8871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:57.219410 8871 hybrid_clock.cc:648] HybridClock initialized: now 1714572537219375 us; error 63 us; skew 500 ppm
I20240501 14:08:57.221720 8871 webserver.cc:460] Webserver started at http://127.7.39.133:33043/ using document root <none> and password file <none>
I20240501 14:08:57.222347 8871 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:57.222431 8871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:57.226301 8871 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:57.228339 8885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:57.229406 8871 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:57.229606 8871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "9faf312806f54f2e8a4a807b234e0d1b"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-rnfw"
I20240501 14:08:57.230080 8871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:57.245595 8871 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:57.246308 8871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:57.246477 8871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:57.247103 8871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:57.248350 8871 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:57.248404 8871 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:57.248446 8871 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:57.248476 8871 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:57.259889 8871 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.133:38805
I20240501 14:08:57.259944 8998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.133:38805 every 8 connection(s)
I20240501 14:08:57.261035 8871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 14:08:57.270402 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 8871
I20240501 14:08:57.270924 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.134:38699
--local_ip_for_outbound_sockets=127.7.39.134
--tserver_master_addrs=127.7.39.190:44225
--webserver_port=38807
--webserver_interface=127.7.39.134
--builtin_ntp_servers=127.7.39.148:46693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 14:08:57.272818 8999 heartbeater.cc:347] Connected to a master server at 127.7.39.190:44225
I20240501 14:08:57.273082 8999 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:57.273833 8999 heartbeater.cc:511] Master 127.7.39.190:44225 requested a full tablet report, sending...
I20240501 14:08:57.275153 7367 ts_manager.cc:194] Re-registered known tserver with Master: 9faf312806f54f2e8a4a807b234e0d1b (127.7.39.133:38805)
I20240501 14:08:57.275776 7367 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.133:36511
W20240501 14:08:57.398548 9002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:57.398844 9002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:57.398933 9002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:57.402079 9002 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:08:57.402156 9002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:57.402256 9002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.134
I20240501 14:08:57.405998 9002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:46693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.134:38699
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.7.39.134
--webserver_port=38807
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.7.39.190:44225
--never_fsync=true
--heap_profile_path=/tmp/kudu.9002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.134
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:57.406927 9002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:57.408147 9002 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:57.417244 9008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:57.418190 9010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:57.419550 9007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:57.420717 9002 server_base.cc:1034] running on GCE node
I20240501 14:08:57.421121 9002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:57.421674 9002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:57.422852 9002 hybrid_clock.cc:648] HybridClock initialized: now 1714572537422836 us; error 37 us; skew 500 ppm
I20240501 14:08:57.425215 9002 webserver.cc:460] Webserver started at http://127.7.39.134:38807/ using document root <none> and password file <none>
I20240501 14:08:57.425853 9002 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:57.425948 9002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:57.429426 9002 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:57.431454 9016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:57.432401 9002 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:57.432511 9002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "9d7c8c0f7907428eab22633702ead7ab"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-rnfw"
I20240501 14:08:57.432940 9002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:57.448829 9002 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:57.449527 9002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:57.449712 9002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:57.450292 9002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:57.451236 9002 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:57.451289 9002 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:57.451351 9002 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:57.451390 9002 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:57.460510 9002 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.134:38699
I20240501 14:08:57.460553 9129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.134:38699 every 8 connection(s)
I20240501 14:08:57.461527 9002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 14:08:57.469944 9130 heartbeater.cc:347] Connected to a master server at 127.7.39.190:44225
I20240501 14:08:57.470170 9130 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:57.470422 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 9002
I20240501 14:08:57.470858 9130 heartbeater.cc:511] Master 127.7.39.190:44225 requested a full tablet report, sending...
I20240501 14:08:57.470911 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.135:39611
--local_ip_for_outbound_sockets=127.7.39.135
--tserver_master_addrs=127.7.39.190:44225
--webserver_port=35631
--webserver_interface=127.7.39.135
--builtin_ntp_servers=127.7.39.148:46693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 14:08:57.472008 7367 ts_manager.cc:194] Re-registered known tserver with Master: 9d7c8c0f7907428eab22633702ead7ab (127.7.39.134:38699)
I20240501 14:08:57.472563 7367 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.134:46523
W20240501 14:08:57.586042 9133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:57.586290 9133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:57.586337 9133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:57.589557 9133 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:08:57.589643 9133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:57.589771 9133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.135
I20240501 14:08:57.593848 9133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:46693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.135:39611
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.7.39.135
--webserver_port=35631
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.7.39.190:44225
--never_fsync=true
--heap_profile_path=/tmp/kudu.9133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.135
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:57.594775 9133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:57.595983 9133 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:57.605578 9141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:57.606549 9138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:57.608038 9139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:57.609174 9133 server_base.cc:1034] running on GCE node
I20240501 14:08:57.609642 9133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:57.610260 9133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:57.611441 9133 hybrid_clock.cc:648] HybridClock initialized: now 1714572537611400 us; error 55 us; skew 500 ppm
I20240501 14:08:57.613821 9133 webserver.cc:460] Webserver started at http://127.7.39.135:35631/ using document root <none> and password file <none>
I20240501 14:08:57.614372 9133 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:57.614447 9133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:57.618000 9133 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:57.619988 9147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:57.621022 9133 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:57.621121 9133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "b4105a6d531d401faadf67b894fec79d"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-rnfw"
I20240501 14:08:57.621529 9133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:57.643747 9133 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:57.644447 9133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:57.644632 9133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:57.645192 9133 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:57.646083 9133 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:57.646129 9133 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:57.646173 9133 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:57.646204 9133 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:57.655120 9133 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.135:39611
I20240501 14:08:57.655131 9260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.135:39611 every 8 connection(s)
I20240501 14:08:57.656183 9133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 14:08:57.658157 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 9133
I20240501 14:08:57.658634 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.136:37475
--local_ip_for_outbound_sockets=127.7.39.136
--tserver_master_addrs=127.7.39.190:44225
--webserver_port=45159
--webserver_interface=127.7.39.136
--builtin_ntp_servers=127.7.39.148:46693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 14:08:57.665660 9261 heartbeater.cc:347] Connected to a master server at 127.7.39.190:44225
I20240501 14:08:57.665867 9261 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:57.666373 9261 heartbeater.cc:511] Master 127.7.39.190:44225 requested a full tablet report, sending...
I20240501 14:08:57.667385 7367 ts_manager.cc:194] Re-registered known tserver with Master: b4105a6d531d401faadf67b894fec79d (127.7.39.135:39611)
I20240501 14:08:57.667904 7367 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.135:33575
W20240501 14:08:57.772614 9264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:57.772883 9264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:57.772950 9264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:57.776052 9264 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:08:57.776122 9264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:57.776213 9264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.136
I20240501 14:08:57.779875 9264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:46693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.136:37475
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.7.39.136
--webserver_port=45159
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.7.39.190:44225
--never_fsync=true
--heap_profile_path=/tmp/kudu.9264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.136
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:57.780829 9264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:57.781953 9264 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:57.791874 9269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:57.793116 9272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:57.794277 9264 server_base.cc:1034] running on GCE node
W20240501 14:08:57.794286 9270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:57.794709 9264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:57.795334 9264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:57.796485 9264 hybrid_clock.cc:648] HybridClock initialized: now 1714572537796472 us; error 29 us; skew 500 ppm
I20240501 14:08:57.798729 9264 webserver.cc:460] Webserver started at http://127.7.39.136:45159/ using document root <none> and password file <none>
I20240501 14:08:57.799432 9264 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:57.799526 9264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:57.803424 9264 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:08:57.805408 9278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:57.806409 9264 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:57.806529 9264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "9d32e843754e44d0b9aae6f472a4d9a2"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-rnfw"
I20240501 14:08:57.806980 9264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:57.821856 9264 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:57.822557 9264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:57.822734 9264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:57.823338 9264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:57.824294 9264 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:57.824340 9264 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:57.824393 9264 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:57.824424 9264 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:57.833122 9264 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.136:37475
I20240501 14:08:57.833149 9391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.136:37475 every 8 connection(s)
I20240501 14:08:57.834163 9264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 14:08:57.834976 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 9264
I20240501 14:08:57.844508 9392 heartbeater.cc:347] Connected to a master server at 127.7.39.190:44225
I20240501 14:08:57.844739 9392 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:57.845269 9392 heartbeater.cc:511] Master 127.7.39.190:44225 requested a full tablet report, sending...
I20240501 14:08:57.846333 7367 ts_manager.cc:194] Re-registered known tserver with Master: 9d32e843754e44d0b9aae6f472a4d9a2 (127.7.39.136:37475)
I20240501 14:08:57.847113 7367 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.136:39151
I20240501 14:08:58.034459 7882 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:58.034785 7752 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:08:58.042788 9064 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:58.046469 7620 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:08:58.047308 9326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:58.058156 8931 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:58.060521 7491 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:58.082981 9195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:58.259886 7360 ts_manager.cc:292] Set tserver state for ae52f888f5094416a24cb3dffbff03c4 to MAINTENANCE_MODE
I20240501 14:08:58.278192 8999 heartbeater.cc:503] Master 127.7.39.190:44225 was elected leader, sending a full tablet report...
W20240501 14:08:58.380170 9456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:58.380447 9456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:58.386234 9456 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:08:58.475135 9130 heartbeater.cc:503] Master 127.7.39.190:44225 was elected leader, sending a full tablet report...
I20240501 14:08:58.481873 9326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:58.484336 7752 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:08:58.484593 7882 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:58.493728 9326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:58.494949 7620 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:08:58.509255 7620 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:08:58.513082 7882 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:58.513074 7752 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:08:58.514454 9195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:58.519223 9064 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:58.523125 7489 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:58.523234 7491 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:58.534438 9195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:58.547339 8931 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:58.552726 9064 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:58.554224 8931 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:58.669790 9261 heartbeater.cc:503] Master 127.7.39.190:44225 was elected leader, sending a full tablet report...
I20240501 14:08:58.675591 8931 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:58.676584 9064 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:58.677981 7752 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:08:58.682241 9326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:58.685034 7620 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:08:58.693974 9195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:58.719309 7491 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:58.724584 7882 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:08:58.779176 9555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:58.779525 9555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:58.782953 9555 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:08:58.849201 9392 heartbeater.cc:503] Master 127.7.39.190:44225 was elected leader, sending a full tablet report...
I20240501 14:08:58.862462 7752 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:08:58.865496 7882 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:58.879808 9195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:58.887457 8931 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:58.889004 7491 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:58.890165 9064 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:58.892284 9326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:58.897022 7620 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:08:58.899245 9326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:58.900532 7491 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:58.912057 7752 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:08:58.916128 8931 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:58.923112 7620 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:08:58.955036 7882 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:58.961261 9195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:58.965740 9064 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:58.996968 7903 consensus_queue.cc:237] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:08:58.998374 7771 raft_consensus.cc:1261] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:08:58.998440 7510 raft_consensus.cc:1261] T c747bd6504c540cebded0019d82495b4 P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:08:58.998828 8567 consensus_queue.cc:1035] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:08:58.999116 9478 consensus_queue.cc:1035] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:08:59.001329 8567 raft_consensus.cc:2932] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } }
I20240501 14:08:59.002863 7771 raft_consensus.cc:2932] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } }
I20240501 14:08:59.004325 7510 raft_consensus.cc:2932] T c747bd6504c540cebded0019d82495b4 P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } }
I20240501 14:08:59.005942 7367 catalog_manager.cc:5484] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "47eabc135e9f4237b036097085e729d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } health_report { overall_health: HEALTHY } } }
I20240501 14:08:59.012184 7903 consensus_queue.cc:237] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: NON_VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: true } }
I20240501 14:08:59.014077 7510 raft_consensus.cc:1261] T c747bd6504c540cebded0019d82495b4 P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:08:59.014448 7771 raft_consensus.cc:1261] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:08:59.014885 9664 consensus_queue.cc:1035] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:08:59.015125 9478 consensus_queue.cc:1035] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:08:59.016767 8567 raft_consensus.cc:2932] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9d32e843754e44d0b9aae6f472a4d9a2 (127.7.39.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: NON_VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: true } } }
I20240501 14:08:59.019021 7354 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c747bd6504c540cebded0019d82495b4 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20240501 14:08:59.019225 7838 consensus_peers.cc:485] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 -> Peer 9d32e843754e44d0b9aae6f472a4d9a2 (127.7.39.136:37475): Couldn't send request to peer 9d32e843754e44d0b9aae6f472a4d9a2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c747bd6504c540cebded0019d82495b4. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:08:59.019501 7369 catalog_manager.cc:5484] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 reported cstate change: config changed from index 4 to 5, NON_VOTER 9d32e843754e44d0b9aae6f472a4d9a2 (127.7.39.136) added. New cstate: current_term: 1 leader_uuid: "47eabc135e9f4237b036097085e729d8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: NON_VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:08:59.019544 7771 raft_consensus.cc:2932] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9d32e843754e44d0b9aae6f472a4d9a2 (127.7.39.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: NON_VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: true } } }
I20240501 14:08:59.019659 7510 raft_consensus.cc:2932] T c747bd6504c540cebded0019d82495b4 P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9d32e843754e44d0b9aae6f472a4d9a2 (127.7.39.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: NON_VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: true } } }
W20240501 14:08:59.133348 9674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:59.133697 9674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:59.137087 9674 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:08:59.216614 9064 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:59.219591 7882 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:59.221841 7752 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:08:59.224625 9326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:59.227675 7620 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:08:59.245394 8931 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:59.245962 7491 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:59.254523 9195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 14:08:59.412066 9717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:59.412328 9717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:59.415346 9717 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:08:59.488179 9748 ts_tablet_manager.cc:902] T c747bd6504c540cebded0019d82495b4 P 9d32e843754e44d0b9aae6f472a4d9a2: Initiating tablet copy from peer 47eabc135e9f4237b036097085e729d8 (127.7.39.132:42369)
I20240501 14:08:59.489266 9748 tablet_copy_client.cc:308] T c747bd6504c540cebded0019d82495b4 P 9d32e843754e44d0b9aae6f472a4d9a2: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.132:42369
I20240501 14:08:59.494943 7923 tablet_copy_service.cc:140] P 47eabc135e9f4237b036097085e729d8: Received BeginTabletCopySession request for tablet c747bd6504c540cebded0019d82495b4 from peer 9d32e843754e44d0b9aae6f472a4d9a2 ({username='slave'} at 127.7.39.136:59617)
I20240501 14:08:59.495149 7923 tablet_copy_service.cc:161] P 47eabc135e9f4237b036097085e729d8: Beginning new tablet copy session on tablet c747bd6504c540cebded0019d82495b4 from peer 9d32e843754e44d0b9aae6f472a4d9a2 at {username='slave'} at 127.7.39.136:59617: session id = 9d32e843754e44d0b9aae6f472a4d9a2-c747bd6504c540cebded0019d82495b4
I20240501 14:08:59.496726 7923 tablet_copy_source_session.cc:205] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:08:59.501259 9748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c747bd6504c540cebded0019d82495b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:59.505463 7882 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:59.508877 9748 tablet_copy_client.cc:783] T c747bd6504c540cebded0019d82495b4 P 9d32e843754e44d0b9aae6f472a4d9a2: tablet copy: Starting download of 0 data blocks...
I20240501 14:08:59.509106 9748 tablet_copy_client.cc:647] T c747bd6504c540cebded0019d82495b4 P 9d32e843754e44d0b9aae6f472a4d9a2: tablet copy: Starting download of 1 WAL segments...
I20240501 14:08:59.509758 7620 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:08:59.509825 8931 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:59.511833 9749 tablet_copy_client.cc:1002] T c747bd6504c540cebded0019d82495b4 P 9d32e843754e44d0b9aae6f472a4d9a2: tablet copy: Injecting latency into file download: 1500
I20240501 14:08:59.522591 9195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:59.523459 7491 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:59.524273 9064 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:59.527563 7752 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:08:59.533493 9326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Runtime error: ksck discovered errors
I20240501 14:08:59.578069 9454 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 7429
W20240501 14:08:59.620364 7839 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111)
W20240501 14:08:59.621218 7839 consensus_peers.cc:485] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:08:59.641078 7839 consensus_peers.cc:485] T 4d282d79e1664f7ea8912e0c5166eec3 P 47eabc135e9f4237b036097085e729d8 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:08:59.772929 7839 consensus_peers.cc:485] T 14e325d8d3184c9192b8cebafb414190 P 47eabc135e9f4237b036097085e729d8 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:08:59.818027 7708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111)
W20240501 14:08:59.818878 7708 consensus_peers.cc:485] T 2880045de45743bdb5358a17d93b2245 P 9ae5a87ca07b40a788a42734b24f66c6 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:08:59.874058 7577 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111)
W20240501 14:08:59.874917 7577 consensus_peers.cc:485] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:08:59.901105 7577 consensus_peers.cc:485] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:08:59.948539 7708 consensus_peers.cc:485] T dcf673c75a8b48da95806e1c43309dda P 9ae5a87ca07b40a788a42734b24f66c6 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:09:00.003959 7839 consensus_peers.cc:485] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:00.660030 9781 raft_consensus.cc:482] T df856440b07b418394477cf368b765c1 P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae52f888f5094416a24cb3dffbff03c4)
I20240501 14:09:00.660169 9781 raft_consensus.cc:504] T df856440b07b418394477cf368b765c1 P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:09:00.660599 9781 leader_election.cc:288] T df856440b07b418394477cf368b765c1 P 47eabc135e9f4237b036097085e729d8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083), 9e44efa72e7d4421b284ae74d1cb9634 (127.7.39.130:46433)
I20240501 14:09:00.661334 7642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df856440b07b418394477cf368b765c1" candidate_uuid: "47eabc135e9f4237b036097085e729d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9e44efa72e7d4421b284ae74d1cb9634" is_pre_election: true
W20240501 14:09:00.661801 7839 leader_election.cc:334] T df856440b07b418394477cf368b765c1 P 47eabc135e9f4237b036097085e729d8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111)
I20240501 14:09:00.661870 7839 leader_election.cc:302] T df856440b07b418394477cf368b765c1 P 47eabc135e9f4237b036097085e729d8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 47eabc135e9f4237b036097085e729d8; no voters: 9e44efa72e7d4421b284ae74d1cb9634, ae52f888f5094416a24cb3dffbff03c4
I20240501 14:09:00.662061 9781 raft_consensus.cc:2726] T df856440b07b418394477cf368b765c1 P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:09:00.675385 9794 raft_consensus.cc:482] T 2cf2f85819584fdf872f6c590d11f873 P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae52f888f5094416a24cb3dffbff03c4)
I20240501 14:09:00.675520 9794 raft_consensus.cc:504] T 2cf2f85819584fdf872f6c590d11f873 P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:09:00.675941 9794 leader_election.cc:288] T 2cf2f85819584fdf872f6c590d11f873 P 9ae5a87ca07b40a788a42734b24f66c6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083), 9e44efa72e7d4421b284ae74d1cb9634 (127.7.39.130:46433)
I20240501 14:09:00.676613 7642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cf2f85819584fdf872f6c590d11f873" candidate_uuid: "9ae5a87ca07b40a788a42734b24f66c6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9e44efa72e7d4421b284ae74d1cb9634" is_pre_election: true
W20240501 14:09:00.677037 7708 leader_election.cc:334] T 2cf2f85819584fdf872f6c590d11f873 P 9ae5a87ca07b40a788a42734b24f66c6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111)
I20240501 14:09:00.677276 7708 leader_election.cc:302] T 2cf2f85819584fdf872f6c590d11f873 P 9ae5a87ca07b40a788a42734b24f66c6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ae5a87ca07b40a788a42734b24f66c6; no voters: 9e44efa72e7d4421b284ae74d1cb9634, ae52f888f5094416a24cb3dffbff03c4
I20240501 14:09:00.677445 9794 raft_consensus.cc:2726] T 2cf2f85819584fdf872f6c590d11f873 P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:09:00.764853 9800 raft_consensus.cc:482] T 2cf2f85819584fdf872f6c590d11f873 P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae52f888f5094416a24cb3dffbff03c4)
I20240501 14:09:00.765029 9800 raft_consensus.cc:504] T 2cf2f85819584fdf872f6c590d11f873 P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:09:00.765429 9800 leader_election.cc:288] T 2cf2f85819584fdf872f6c590d11f873 P 9e44efa72e7d4421b284ae74d1cb9634 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083), 9ae5a87ca07b40a788a42734b24f66c6 (127.7.39.131:34697)
I20240501 14:09:00.766120 7772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cf2f85819584fdf872f6c590d11f873" candidate_uuid: "9e44efa72e7d4421b284ae74d1cb9634" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9ae5a87ca07b40a788a42734b24f66c6" is_pre_election: true
I20240501 14:09:00.766287 7772 raft_consensus.cc:2454] T 2cf2f85819584fdf872f6c590d11f873 P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e44efa72e7d4421b284ae74d1cb9634 in term 1.
W20240501 14:09:00.766551 7577 leader_election.cc:334] T 2cf2f85819584fdf872f6c590d11f873 P 9e44efa72e7d4421b284ae74d1cb9634 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111)
I20240501 14:09:00.766718 7577 leader_election.cc:302] T 2cf2f85819584fdf872f6c590d11f873 P 9e44efa72e7d4421b284ae74d1cb9634 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9ae5a87ca07b40a788a42734b24f66c6, 9e44efa72e7d4421b284ae74d1cb9634; no voters: ae52f888f5094416a24cb3dffbff03c4
I20240501 14:09:00.766872 9800 raft_consensus.cc:2781] T 2cf2f85819584fdf872f6c590d11f873 P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:09:00.766937 9800 raft_consensus.cc:482] T 2cf2f85819584fdf872f6c590d11f873 P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ae52f888f5094416a24cb3dffbff03c4)
I20240501 14:09:00.766979 9800 raft_consensus.cc:3037] T 2cf2f85819584fdf872f6c590d11f873 P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:00.767920 9800 raft_consensus.cc:504] T 2cf2f85819584fdf872f6c590d11f873 P 9e44efa72e7d4421b284ae74d1cb9634 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:09:00.768256 9800 leader_election.cc:288] T 2cf2f85819584fdf872f6c590d11f873 P 9e44efa72e7d4421b284ae74d1cb9634 [CANDIDATE]: Term 2 election: Requested vote from peers ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083), 9ae5a87ca07b40a788a42734b24f66c6 (127.7.39.131:34697)
I20240501 14:09:00.768852 7772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cf2f85819584fdf872f6c590d11f873" candidate_uuid: "9e44efa72e7d4421b284ae74d1cb9634" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9ae5a87ca07b40a788a42734b24f66c6"
I20240501 14:09:00.769006 7772 raft_consensus.cc:3037] T 2cf2f85819584fdf872f6c590d11f873 P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:09:00.769135 7577 leader_election.cc:334] T 2cf2f85819584fdf872f6c590d11f873 P 9e44efa72e7d4421b284ae74d1cb9634 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111)
I20240501 14:09:00.769904 7772 raft_consensus.cc:2454] T 2cf2f85819584fdf872f6c590d11f873 P 9ae5a87ca07b40a788a42734b24f66c6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e44efa72e7d4421b284ae74d1cb9634 in term 2.
I20240501 14:09:00.770186 7577 leader_election.cc:302] T 2cf2f85819584fdf872f6c590d11f873 P 9e44efa72e7d4421b284ae74d1cb9634 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9ae5a87ca07b40a788a42734b24f66c6, 9e44efa72e7d4421b284ae74d1cb9634; no voters: ae52f888f5094416a24cb3dffbff03c4
I20240501 14:09:00.770334 9800 raft_consensus.cc:2781] T 2cf2f85819584fdf872f6c590d11f873 P 9e44efa72e7d4421b284ae74d1cb9634 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:00.770450 9800 raft_consensus.cc:686] T 2cf2f85819584fdf872f6c590d11f873 P 9e44efa72e7d4421b284ae74d1cb9634 [term 2 LEADER]: Becoming Leader. State: Replica: 9e44efa72e7d4421b284ae74d1cb9634, State: Running, Role: LEADER
I20240501 14:09:00.770593 9800 consensus_queue.cc:237] T 2cf2f85819584fdf872f6c590d11f873 P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:09:00.772303 7364 catalog_manager.cc:5484] T 2cf2f85819584fdf872f6c590d11f873 P 9e44efa72e7d4421b284ae74d1cb9634 reported cstate change: term changed from 1 to 2, leader changed from ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129) to 9e44efa72e7d4421b284ae74d1cb9634 (127.7.39.130). New cstate: current_term: 2 leader_uuid: "9e44efa72e7d4421b284ae74d1cb9634" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:00.802268 9781 raft_consensus.cc:482] T ceee52d679bf4d8e8896a6c0dde0519b P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae52f888f5094416a24cb3dffbff03c4)
I20240501 14:09:00.802407 9781 raft_consensus.cc:504] T ceee52d679bf4d8e8896a6c0dde0519b P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:09:00.802824 9781 leader_election.cc:288] T ceee52d679bf4d8e8896a6c0dde0519b P 47eabc135e9f4237b036097085e729d8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083), 9ae5a87ca07b40a788a42734b24f66c6 (127.7.39.131:34697)
I20240501 14:09:00.803597 7772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceee52d679bf4d8e8896a6c0dde0519b" candidate_uuid: "47eabc135e9f4237b036097085e729d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9ae5a87ca07b40a788a42734b24f66c6" is_pre_election: true
W20240501 14:09:00.803966 7839 leader_election.cc:334] T ceee52d679bf4d8e8896a6c0dde0519b P 47eabc135e9f4237b036097085e729d8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111)
I20240501 14:09:00.804112 7839 leader_election.cc:302] T ceee52d679bf4d8e8896a6c0dde0519b P 47eabc135e9f4237b036097085e729d8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 47eabc135e9f4237b036097085e729d8; no voters: 9ae5a87ca07b40a788a42734b24f66c6, ae52f888f5094416a24cb3dffbff03c4
I20240501 14:09:00.804251 9781 raft_consensus.cc:2726] T ceee52d679bf4d8e8896a6c0dde0519b P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:09:00.920908 9800 raft_consensus.cc:482] T 3137e7904ad04fc79d4028c1882e2412 P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae52f888f5094416a24cb3dffbff03c4)
I20240501 14:09:00.921084 9800 raft_consensus.cc:504] T 3137e7904ad04fc79d4028c1882e2412 P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } }
I20240501 14:09:00.921572 9800 leader_election.cc:288] T 3137e7904ad04fc79d4028c1882e2412 P 9e44efa72e7d4421b284ae74d1cb9634 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083), 47eabc135e9f4237b036097085e729d8 (127.7.39.132:42369)
I20240501 14:09:00.922122 7903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3137e7904ad04fc79d4028c1882e2412" candidate_uuid: "9e44efa72e7d4421b284ae74d1cb9634" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "47eabc135e9f4237b036097085e729d8" is_pre_election: true
W20240501 14:09:00.922816 7577 leader_election.cc:334] T 3137e7904ad04fc79d4028c1882e2412 P 9e44efa72e7d4421b284ae74d1cb9634 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111)
I20240501 14:09:00.922889 7577 leader_election.cc:302] T 3137e7904ad04fc79d4028c1882e2412 P 9e44efa72e7d4421b284ae74d1cb9634 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e44efa72e7d4421b284ae74d1cb9634; no voters: 47eabc135e9f4237b036097085e729d8, ae52f888f5094416a24cb3dffbff03c4
I20240501 14:09:00.923105 9800 raft_consensus.cc:2726] T 3137e7904ad04fc79d4028c1882e2412 P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:09:00.957237 9794 raft_consensus.cc:482] T ceee52d679bf4d8e8896a6c0dde0519b P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae52f888f5094416a24cb3dffbff03c4)
I20240501 14:09:00.957392 9794 raft_consensus.cc:504] T ceee52d679bf4d8e8896a6c0dde0519b P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:09:00.957918 9794 leader_election.cc:288] T ceee52d679bf4d8e8896a6c0dde0519b P 9ae5a87ca07b40a788a42734b24f66c6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 47eabc135e9f4237b036097085e729d8 (127.7.39.132:42369), ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083)
I20240501 14:09:00.958359 7903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceee52d679bf4d8e8896a6c0dde0519b" candidate_uuid: "9ae5a87ca07b40a788a42734b24f66c6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "47eabc135e9f4237b036097085e729d8" is_pre_election: true
I20240501 14:09:00.958539 7903 raft_consensus.cc:2454] T ceee52d679bf4d8e8896a6c0dde0519b P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ae5a87ca07b40a788a42734b24f66c6 in term 1.
I20240501 14:09:00.958859 7707 leader_election.cc:302] T ceee52d679bf4d8e8896a6c0dde0519b P 9ae5a87ca07b40a788a42734b24f66c6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47eabc135e9f4237b036097085e729d8, 9ae5a87ca07b40a788a42734b24f66c6; no voters:
I20240501 14:09:00.959059 9794 raft_consensus.cc:2781] T ceee52d679bf4d8e8896a6c0dde0519b P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240501 14:09:00.959081 7708 leader_election.cc:334] T ceee52d679bf4d8e8896a6c0dde0519b P 9ae5a87ca07b40a788a42734b24f66c6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111)
I20240501 14:09:00.959161 9794 raft_consensus.cc:482] T ceee52d679bf4d8e8896a6c0dde0519b P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ae52f888f5094416a24cb3dffbff03c4)
I20240501 14:09:00.959218 9794 raft_consensus.cc:3037] T ceee52d679bf4d8e8896a6c0dde0519b P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:00.960096 9794 raft_consensus.cc:504] T ceee52d679bf4d8e8896a6c0dde0519b P 9ae5a87ca07b40a788a42734b24f66c6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:09:00.960417 9794 leader_election.cc:288] T ceee52d679bf4d8e8896a6c0dde0519b P 9ae5a87ca07b40a788a42734b24f66c6 [CANDIDATE]: Term 2 election: Requested vote from peers 47eabc135e9f4237b036097085e729d8 (127.7.39.132:42369), ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083)
I20240501 14:09:00.960769 7903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceee52d679bf4d8e8896a6c0dde0519b" candidate_uuid: "9ae5a87ca07b40a788a42734b24f66c6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "47eabc135e9f4237b036097085e729d8"
I20240501 14:09:00.960893 7903 raft_consensus.cc:3037] T ceee52d679bf4d8e8896a6c0dde0519b P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:09:00.961347 7708 leader_election.cc:334] T ceee52d679bf4d8e8896a6c0dde0519b P 9ae5a87ca07b40a788a42734b24f66c6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111)
I20240501 14:09:00.961880 7903 raft_consensus.cc:2454] T ceee52d679bf4d8e8896a6c0dde0519b P 47eabc135e9f4237b036097085e729d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ae5a87ca07b40a788a42734b24f66c6 in term 2.
I20240501 14:09:00.962206 7707 leader_election.cc:302] T ceee52d679bf4d8e8896a6c0dde0519b P 9ae5a87ca07b40a788a42734b24f66c6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 47eabc135e9f4237b036097085e729d8, 9ae5a87ca07b40a788a42734b24f66c6; no voters: ae52f888f5094416a24cb3dffbff03c4
I20240501 14:09:00.962373 9794 raft_consensus.cc:2781] T ceee52d679bf4d8e8896a6c0dde0519b P 9ae5a87ca07b40a788a42734b24f66c6 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:00.962553 9794 raft_consensus.cc:686] T ceee52d679bf4d8e8896a6c0dde0519b P 9ae5a87ca07b40a788a42734b24f66c6 [term 2 LEADER]: Becoming Leader. State: Replica: 9ae5a87ca07b40a788a42734b24f66c6, State: Running, Role: LEADER
I20240501 14:09:00.962699 9794 consensus_queue.cc:237] T ceee52d679bf4d8e8896a6c0dde0519b P 9ae5a87ca07b40a788a42734b24f66c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:09:00.964421 7364 catalog_manager.cc:5484] T ceee52d679bf4d8e8896a6c0dde0519b P 9ae5a87ca07b40a788a42734b24f66c6 reported cstate change: term changed from 1 to 2, leader changed from ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129) to 9ae5a87ca07b40a788a42734b24f66c6 (127.7.39.131). New cstate: current_term: 2 leader_uuid: "9ae5a87ca07b40a788a42734b24f66c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:01.012336 9748 tablet_copy_client.cc:523] T c747bd6504c540cebded0019d82495b4 P 9d32e843754e44d0b9aae6f472a4d9a2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:01.014313 9748 tablet_bootstrap.cc:492] T c747bd6504c540cebded0019d82495b4 P 9d32e843754e44d0b9aae6f472a4d9a2: Bootstrap starting.
I20240501 14:09:01.020448 9748 log.cc:824] T c747bd6504c540cebded0019d82495b4 P 9d32e843754e44d0b9aae6f472a4d9a2: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:01.039834 9781 raft_consensus.cc:482] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae52f888f5094416a24cb3dffbff03c4)
I20240501 14:09:01.039960 9781 raft_consensus.cc:504] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } }
I20240501 14:09:01.040364 9781 leader_election.cc:288] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9e44efa72e7d4421b284ae74d1cb9634 (127.7.39.130:46433), ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083)
I20240501 14:09:01.040930 7642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3137e7904ad04fc79d4028c1882e2412" candidate_uuid: "47eabc135e9f4237b036097085e729d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9e44efa72e7d4421b284ae74d1cb9634" is_pre_election: true
I20240501 14:09:01.041078 7642 raft_consensus.cc:2454] T 3137e7904ad04fc79d4028c1882e2412 P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47eabc135e9f4237b036097085e729d8 in term 1.
I20240501 14:09:01.041461 7839 leader_election.cc:302] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47eabc135e9f4237b036097085e729d8, 9e44efa72e7d4421b284ae74d1cb9634; no voters:
W20240501 14:09:01.041610 7839 leader_election.cc:334] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111)
I20240501 14:09:01.041661 9781 raft_consensus.cc:2781] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:09:01.041712 9781 raft_consensus.cc:482] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ae52f888f5094416a24cb3dffbff03c4)
I20240501 14:09:01.041752 9781 raft_consensus.cc:3037] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:01.041803 9748 tablet_bootstrap.cc:492] T c747bd6504c540cebded0019d82495b4 P 9d32e843754e44d0b9aae6f472a4d9a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:01.042137 9748 tablet_bootstrap.cc:492] T c747bd6504c540cebded0019d82495b4 P 9d32e843754e44d0b9aae6f472a4d9a2: Bootstrap complete.
I20240501 14:09:01.042439 9748 ts_tablet_manager.cc:1367] T c747bd6504c540cebded0019d82495b4 P 9d32e843754e44d0b9aae6f472a4d9a2: Time spent bootstrapping tablet: real 0.028s user 0.027s sys 0.000s
I20240501 14:09:01.042660 9781 raft_consensus.cc:504] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } }
I20240501 14:09:01.042964 9781 leader_election.cc:288] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 [CANDIDATE]: Term 2 election: Requested vote from peers 9e44efa72e7d4421b284ae74d1cb9634 (127.7.39.130:46433), ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083)
I20240501 14:09:01.043295 7642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3137e7904ad04fc79d4028c1882e2412" candidate_uuid: "47eabc135e9f4237b036097085e729d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9e44efa72e7d4421b284ae74d1cb9634"
I20240501 14:09:01.043426 7642 raft_consensus.cc:3037] T 3137e7904ad04fc79d4028c1882e2412 P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:09:01.044031 7839 leader_election.cc:334] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111)
I20240501 14:09:01.044437 7642 raft_consensus.cc:2454] T 3137e7904ad04fc79d4028c1882e2412 P 9e44efa72e7d4421b284ae74d1cb9634 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47eabc135e9f4237b036097085e729d8 in term 2.
I20240501 14:09:01.044708 7839 leader_election.cc:302] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 47eabc135e9f4237b036097085e729d8, 9e44efa72e7d4421b284ae74d1cb9634; no voters: ae52f888f5094416a24cb3dffbff03c4
I20240501 14:09:01.044852 9781 raft_consensus.cc:2781] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:01.044760 9748 raft_consensus.cc:348] T c747bd6504c540cebded0019d82495b4 P 9d32e843754e44d0b9aae6f472a4d9a2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: NON_VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: true } }
I20240501 14:09:01.045022 9781 raft_consensus.cc:686] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 [term 2 LEADER]: Becoming Leader. State: Replica: 47eabc135e9f4237b036097085e729d8, State: Running, Role: LEADER
I20240501 14:09:01.045042 9748 raft_consensus.cc:729] T c747bd6504c540cebded0019d82495b4 P 9d32e843754e44d0b9aae6f472a4d9a2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9d32e843754e44d0b9aae6f472a4d9a2, State: Initialized, Role: LEARNER
I20240501 14:09:01.045169 9781 consensus_queue.cc:237] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } }
I20240501 14:09:01.045405 9748 consensus_queue.cc:260] T c747bd6504c540cebded0019d82495b4 P 9d32e843754e44d0b9aae6f472a4d9a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: NON_VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: true } }
I20240501 14:09:01.046321 9748 ts_tablet_manager.cc:1398] T c747bd6504c540cebded0019d82495b4 P 9d32e843754e44d0b9aae6f472a4d9a2: Time spent starting tablet: real 0.004s user 0.007s sys 0.000s
I20240501 14:09:01.046957 7923 tablet_copy_service.cc:342] P 47eabc135e9f4237b036097085e729d8: Request end of tablet copy session 9d32e843754e44d0b9aae6f472a4d9a2-c747bd6504c540cebded0019d82495b4 received from {username='slave'} at 127.7.39.136:59617
I20240501 14:09:01.046869 7364 catalog_manager.cc:5484] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 reported cstate change: term changed from 1 to 2, leader changed from ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129) to 47eabc135e9f4237b036097085e729d8 (127.7.39.132). New cstate: current_term: 2 leader_uuid: "47eabc135e9f4237b036097085e729d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:01.047060 7923 tablet_copy_service.cc:433] P 47eabc135e9f4237b036097085e729d8: ending tablet copy session 9d32e843754e44d0b9aae6f472a4d9a2-c747bd6504c540cebded0019d82495b4 on tablet c747bd6504c540cebded0019d82495b4 with peer 9d32e843754e44d0b9aae6f472a4d9a2
I20240501 14:09:01.056942 9800 raft_consensus.cc:482] T df856440b07b418394477cf368b765c1 P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae52f888f5094416a24cb3dffbff03c4)
I20240501 14:09:01.057075 9800 raft_consensus.cc:504] T df856440b07b418394477cf368b765c1 P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:09:01.057456 9800 leader_election.cc:288] T df856440b07b418394477cf368b765c1 P 9e44efa72e7d4421b284ae74d1cb9634 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 47eabc135e9f4237b036097085e729d8 (127.7.39.132:42369), ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083)
I20240501 14:09:01.057864 7903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df856440b07b418394477cf368b765c1" candidate_uuid: "9e44efa72e7d4421b284ae74d1cb9634" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "47eabc135e9f4237b036097085e729d8" is_pre_election: true
I20240501 14:09:01.058018 7903 raft_consensus.cc:2454] T df856440b07b418394477cf368b765c1 P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e44efa72e7d4421b284ae74d1cb9634 in term 1.
I20240501 14:09:01.058333 7576 leader_election.cc:302] T df856440b07b418394477cf368b765c1 P 9e44efa72e7d4421b284ae74d1cb9634 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47eabc135e9f4237b036097085e729d8, 9e44efa72e7d4421b284ae74d1cb9634; no voters:
I20240501 14:09:01.058509 9800 raft_consensus.cc:2781] T df856440b07b418394477cf368b765c1 P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:09:01.058565 9800 raft_consensus.cc:482] T df856440b07b418394477cf368b765c1 P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ae52f888f5094416a24cb3dffbff03c4)
I20240501 14:09:01.058594 9800 raft_consensus.cc:3037] T df856440b07b418394477cf368b765c1 P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:09:01.058640 7577 leader_election.cc:334] T df856440b07b418394477cf368b765c1 P 9e44efa72e7d4421b284ae74d1cb9634 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111)
I20240501 14:09:01.059671 9800 raft_consensus.cc:504] T df856440b07b418394477cf368b765c1 P 9e44efa72e7d4421b284ae74d1cb9634 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:09:01.059986 9800 leader_election.cc:288] T df856440b07b418394477cf368b765c1 P 9e44efa72e7d4421b284ae74d1cb9634 [CANDIDATE]: Term 2 election: Requested vote from peers 47eabc135e9f4237b036097085e729d8 (127.7.39.132:42369), ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083)
I20240501 14:09:01.060462 7903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df856440b07b418394477cf368b765c1" candidate_uuid: "9e44efa72e7d4421b284ae74d1cb9634" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "47eabc135e9f4237b036097085e729d8"
I20240501 14:09:01.060566 7903 raft_consensus.cc:3037] T df856440b07b418394477cf368b765c1 P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:09:01.060928 7577 leader_election.cc:334] T df856440b07b418394477cf368b765c1 P 9e44efa72e7d4421b284ae74d1cb9634 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111)
I20240501 14:09:01.061477 7903 raft_consensus.cc:2454] T df856440b07b418394477cf368b765c1 P 47eabc135e9f4237b036097085e729d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e44efa72e7d4421b284ae74d1cb9634 in term 2.
I20240501 14:09:01.061762 7576 leader_election.cc:302] T df856440b07b418394477cf368b765c1 P 9e44efa72e7d4421b284ae74d1cb9634 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 47eabc135e9f4237b036097085e729d8, 9e44efa72e7d4421b284ae74d1cb9634; no voters: ae52f888f5094416a24cb3dffbff03c4
I20240501 14:09:01.061985 9800 raft_consensus.cc:2781] T df856440b07b418394477cf368b765c1 P 9e44efa72e7d4421b284ae74d1cb9634 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:01.062073 9800 raft_consensus.cc:686] T df856440b07b418394477cf368b765c1 P 9e44efa72e7d4421b284ae74d1cb9634 [term 2 LEADER]: Becoming Leader. State: Replica: 9e44efa72e7d4421b284ae74d1cb9634, State: Running, Role: LEADER
I20240501 14:09:01.062216 9800 consensus_queue.cc:237] T df856440b07b418394477cf368b765c1 P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:09:01.063822 7364 catalog_manager.cc:5484] T df856440b07b418394477cf368b765c1 P 9e44efa72e7d4421b284ae74d1cb9634 reported cstate change: term changed from 1 to 2, leader changed from ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129) to 9e44efa72e7d4421b284ae74d1cb9634 (127.7.39.130). New cstate: current_term: 2 leader_uuid: "9e44efa72e7d4421b284ae74d1cb9634" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:01.161438 7772 raft_consensus.cc:1261] T 2cf2f85819584fdf872f6c590d11f873 P 9ae5a87ca07b40a788a42734b24f66c6 [term 2 FOLLOWER]: Refusing update from remote peer 9e44efa72e7d4421b284ae74d1cb9634: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:09:01.161950 9800 consensus_queue.cc:1035] T 2cf2f85819584fdf872f6c590d11f873 P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:09:01.165416 7577 consensus_peers.cc:485] T 2cf2f85819584fdf872f6c590d11f873 P 9e44efa72e7d4421b284ae74d1cb9634 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:01.392364 7903 raft_consensus.cc:1261] T ceee52d679bf4d8e8896a6c0dde0519b P 47eabc135e9f4237b036097085e729d8 [term 2 FOLLOWER]: Refusing update from remote peer 9ae5a87ca07b40a788a42734b24f66c6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:09:01.392803 9794 consensus_queue.cc:1035] T ceee52d679bf4d8e8896a6c0dde0519b P 9ae5a87ca07b40a788a42734b24f66c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:09:01.396349 7708 consensus_peers.cc:485] T ceee52d679bf4d8e8896a6c0dde0519b P 9ae5a87ca07b40a788a42734b24f66c6 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:09:01.431396 7839 consensus_peers.cc:485] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:01.482535 7642 raft_consensus.cc:1261] T 3137e7904ad04fc79d4028c1882e2412 P 9e44efa72e7d4421b284ae74d1cb9634 [term 2 FOLLOWER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:09:01.482982 9781 consensus_queue.cc:1035] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:01.565639 7903 raft_consensus.cc:1261] T df856440b07b418394477cf368b765c1 P 47eabc135e9f4237b036097085e729d8 [term 2 FOLLOWER]: Refusing update from remote peer 9e44efa72e7d4421b284ae74d1cb9634: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:09:01.566174 9800 consensus_queue.cc:1035] T df856440b07b418394477cf368b765c1 P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:09:01.569782 7577 consensus_peers.cc:485] T df856440b07b418394477cf368b765c1 P 9e44efa72e7d4421b284ae74d1cb9634 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:01.577937 9346 raft_consensus.cc:1203] T c747bd6504c540cebded0019d82495b4 P 9d32e843754e44d0b9aae6f472a4d9a2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
W20240501 14:09:01.933215 7839 consensus_peers.cc:485] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:09:02.105906 7839 consensus_peers.cc:485] T 4d282d79e1664f7ea8912e0c5166eec3 P 47eabc135e9f4237b036097085e729d8 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:09:02.121023 7708 consensus_peers.cc:485] T dcf673c75a8b48da95806e1c43309dda P 9ae5a87ca07b40a788a42734b24f66c6 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:09:02.132222 9857 raft_consensus.cc:1050] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8: attempting to promote NON_VOTER 9d32e843754e44d0b9aae6f472a4d9a2 to VOTER
I20240501 14:09:02.132681 9857 consensus_queue.cc:237] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } }
I20240501 14:09:02.134233 9346 raft_consensus.cc:1261] T c747bd6504c540cebded0019d82495b4 P 9d32e843754e44d0b9aae6f472a4d9a2 [term 1 LEARNER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:02.134980 7772 raft_consensus.cc:1261] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:02.134936 9859 consensus_queue.cc:1035] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:02.135771 9781 consensus_queue.cc:1035] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 14:09:02.135880 7839 consensus_peers.cc:485] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:02.137990 9781 raft_consensus.cc:2932] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9d32e843754e44d0b9aae6f472a4d9a2 (127.7.39.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } }
I20240501 14:09:02.138514 9346 raft_consensus.cc:2932] T c747bd6504c540cebded0019d82495b4 P 9d32e843754e44d0b9aae6f472a4d9a2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9d32e843754e44d0b9aae6f472a4d9a2 (127.7.39.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } }
I20240501 14:09:02.141530 7364 catalog_manager.cc:5484] T c747bd6504c540cebded0019d82495b4 P 9d32e843754e44d0b9aae6f472a4d9a2 reported cstate change: config changed from index 5 to 6, 9d32e843754e44d0b9aae6f472a4d9a2 (127.7.39.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "47eabc135e9f4237b036097085e729d8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } }
I20240501 14:09:02.141654 7772 raft_consensus.cc:2932] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9d32e843754e44d0b9aae6f472a4d9a2 (127.7.39.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } }
I20240501 14:09:02.145385 7904 consensus_queue.cc:237] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } }
I20240501 14:09:02.146437 7772 raft_consensus.cc:1261] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:02.146466 9346 raft_consensus.cc:1261] T c747bd6504c540cebded0019d82495b4 P 9d32e843754e44d0b9aae6f472a4d9a2 [term 1 FOLLOWER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:02.146833 9781 consensus_queue.cc:1035] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:02.147001 9859 consensus_queue.cc:1035] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:02.148425 9781 raft_consensus.cc:2932] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } }
I20240501 14:09:02.148963 7772 raft_consensus.cc:2932] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } }
I20240501 14:09:02.149235 9346 raft_consensus.cc:2932] T c747bd6504c540cebded0019d82495b4 P 9d32e843754e44d0b9aae6f472a4d9a2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } }
I20240501 14:09:02.150337 7354 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c747bd6504c540cebded0019d82495b4 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:02.151517 7365 catalog_manager.cc:5484] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 reported cstate change: config changed from index 6 to 7, VOTER ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129) evicted. New cstate: current_term: 1 leader_uuid: "47eabc135e9f4237b036097085e729d8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } }
W20240501 14:09:02.154889 7355 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111)
W20240501 14:09:02.155858 7355 catalog_manager.cc:4573] TS ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c747bd6504c540cebded0019d82495b4: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111)
W20240501 14:09:02.229166 7577 consensus_peers.cc:485] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:09:02.424711 7839 consensus_peers.cc:485] T 14e325d8d3184c9192b8cebafb414190 P 47eabc135e9f4237b036097085e729d8 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:09:02.428515 7708 consensus_peers.cc:485] T 2880045de45743bdb5358a17d93b2245 P 9ae5a87ca07b40a788a42734b24f66c6 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:09:02.554606 7904 consensus_queue.cc:237] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } }
I20240501 14:09:02.556159 7642 raft_consensus.cc:1261] T 3137e7904ad04fc79d4028c1882e2412 P 9e44efa72e7d4421b284ae74d1cb9634 [term 2 FOLLOWER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20240501 14:09:02.556414 7839 consensus_peers.cc:485] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:02.556648 9859 consensus_queue.cc:1035] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:02.558393 9860 raft_consensus.cc:2932] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index -1 to 5. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } }
I20240501 14:09:02.558825 7642 raft_consensus.cc:2932] T 3137e7904ad04fc79d4028c1882e2412 P 9e44efa72e7d4421b284ae74d1cb9634 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index -1 to 5. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } }
I20240501 14:09:02.590828 7366 catalog_manager.cc:5484] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 reported cstate change: config changed from index -1 to 5. New cstate: current_term: 2 leader_uuid: "47eabc135e9f4237b036097085e729d8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:02.594661 7904 consensus_queue.cc:237] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: NON_VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: true } }
I20240501 14:09:02.596850 7642 raft_consensus.cc:1261] T 3137e7904ad04fc79d4028c1882e2412 P 9e44efa72e7d4421b284ae74d1cb9634 [term 2 FOLLOWER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:09:02.597337 9860 consensus_queue.cc:1035] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 14:09:02.599110 7839 consensus_peers.cc:485] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:02.600051 9859 raft_consensus.cc:2932] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 5 to 6, NON_VOTER b4105a6d531d401faadf67b894fec79d (127.7.39.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: NON_VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: true } } }
I20240501 14:09:02.600633 7642 raft_consensus.cc:2932] T 3137e7904ad04fc79d4028c1882e2412 P 9e44efa72e7d4421b284ae74d1cb9634 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 5 to 6, NON_VOTER b4105a6d531d401faadf67b894fec79d (127.7.39.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: NON_VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: true } } }
W20240501 14:09:02.602221 7839 consensus_peers.cc:485] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 -> Peer b4105a6d531d401faadf67b894fec79d (127.7.39.135:39611): Couldn't send request to peer b4105a6d531d401faadf67b894fec79d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3137e7904ad04fc79d4028c1882e2412. This is attempt 1: this message will repeat every 5th retry.
W20240501 14:09:02.657359 7577 consensus_peers.cc:485] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:09:02.689392 7354 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3137e7904ad04fc79d4028c1882e2412 with cas_config_opid_index 5: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:09:02.690685 7366 catalog_manager.cc:5484] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 reported cstate change: config changed from index 5 to 6, NON_VOTER b4105a6d531d401faadf67b894fec79d (127.7.39.135) added. New cstate: current_term: 2 leader_uuid: "47eabc135e9f4237b036097085e729d8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: NON_VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:03.079691 9904 ts_tablet_manager.cc:902] T 3137e7904ad04fc79d4028c1882e2412 P b4105a6d531d401faadf67b894fec79d: Initiating tablet copy from peer 47eabc135e9f4237b036097085e729d8 (127.7.39.132:42369)
I20240501 14:09:03.080449 9904 tablet_copy_client.cc:308] T 3137e7904ad04fc79d4028c1882e2412 P b4105a6d531d401faadf67b894fec79d: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.132:42369
I20240501 14:09:03.084340 7923 tablet_copy_service.cc:140] P 47eabc135e9f4237b036097085e729d8: Received BeginTabletCopySession request for tablet 3137e7904ad04fc79d4028c1882e2412 from peer b4105a6d531d401faadf67b894fec79d ({username='slave'} at 127.7.39.135:36319)
I20240501 14:09:03.084466 7923 tablet_copy_service.cc:161] P 47eabc135e9f4237b036097085e729d8: Beginning new tablet copy session on tablet 3137e7904ad04fc79d4028c1882e2412 from peer b4105a6d531d401faadf67b894fec79d at {username='slave'} at 127.7.39.135:36319: session id = b4105a6d531d401faadf67b894fec79d-3137e7904ad04fc79d4028c1882e2412
I20240501 14:09:03.085194 7923 tablet_copy_source_session.cc:205] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:03.088243 9904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3137e7904ad04fc79d4028c1882e2412. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:03.093436 9904 tablet_copy_client.cc:783] T 3137e7904ad04fc79d4028c1882e2412 P b4105a6d531d401faadf67b894fec79d: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:03.093641 9904 tablet_copy_client.cc:647] T 3137e7904ad04fc79d4028c1882e2412 P b4105a6d531d401faadf67b894fec79d: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:03.094964 9908 tablet_copy_client.cc:1002] T 3137e7904ad04fc79d4028c1882e2412 P b4105a6d531d401faadf67b894fec79d: tablet copy: Injecting latency into file download: 1500
W20240501 14:09:03.366467 7355 catalog_manager.cc:4573] TS ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c747bd6504c540cebded0019d82495b4: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111)
W20240501 14:09:03.515405 7577 consensus_peers.cc:485] T 2cf2f85819584fdf872f6c590d11f873 P 9e44efa72e7d4421b284ae74d1cb9634 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:09:03.872676 7708 consensus_peers.cc:485] T ceee52d679bf4d8e8896a6c0dde0519b P 9ae5a87ca07b40a788a42734b24f66c6 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:09:04.344676 7839 consensus_peers.cc:485] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:09:04.372821 7577 consensus_peers.cc:485] T df856440b07b418394477cf368b765c1 P 9e44efa72e7d4421b284ae74d1cb9634 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:09:04.434631 7355 catalog_manager.cc:4573] TS ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c747bd6504c540cebded0019d82495b4: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111)
W20240501 14:09:04.513710 7839 consensus_peers.cc:485] T 4d282d79e1664f7ea8912e0c5166eec3 P 47eabc135e9f4237b036097085e729d8 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:09:04.595400 9904 tablet_copy_client.cc:523] T 3137e7904ad04fc79d4028c1882e2412 P b4105a6d531d401faadf67b894fec79d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:04.597458 9904 tablet_bootstrap.cc:492] T 3137e7904ad04fc79d4028c1882e2412 P b4105a6d531d401faadf67b894fec79d: Bootstrap starting.
I20240501 14:09:04.602792 9904 log.cc:824] T 3137e7904ad04fc79d4028c1882e2412 P b4105a6d531d401faadf67b894fec79d: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:04.621372 9904 tablet_bootstrap.cc:492] T 3137e7904ad04fc79d4028c1882e2412 P b4105a6d531d401faadf67b894fec79d: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:04.621729 9904 tablet_bootstrap.cc:492] T 3137e7904ad04fc79d4028c1882e2412 P b4105a6d531d401faadf67b894fec79d: Bootstrap complete.
W20240501 14:09:04.621802 7839 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240501 14:09:04.622094 9904 ts_tablet_manager.cc:1367] T 3137e7904ad04fc79d4028c1882e2412 P b4105a6d531d401faadf67b894fec79d: Time spent bootstrapping tablet: real 0.025s user 0.019s sys 0.004s
I20240501 14:09:04.624245 9904 raft_consensus.cc:348] T 3137e7904ad04fc79d4028c1882e2412 P b4105a6d531d401faadf67b894fec79d [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: NON_VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: true } }
I20240501 14:09:04.624457 9904 raft_consensus.cc:729] T 3137e7904ad04fc79d4028c1882e2412 P b4105a6d531d401faadf67b894fec79d [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: b4105a6d531d401faadf67b894fec79d, State: Initialized, Role: LEARNER
I20240501 14:09:04.624759 9904 consensus_queue.cc:260] T 3137e7904ad04fc79d4028c1882e2412 P b4105a6d531d401faadf67b894fec79d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: NON_VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: true } }
I20240501 14:09:04.625523 9904 ts_tablet_manager.cc:1398] T 3137e7904ad04fc79d4028c1882e2412 P b4105a6d531d401faadf67b894fec79d: Time spent starting tablet: real 0.003s user 0.006s sys 0.000s
I20240501 14:09:04.626235 7923 tablet_copy_service.cc:342] P 47eabc135e9f4237b036097085e729d8: Request end of tablet copy session b4105a6d531d401faadf67b894fec79d-3137e7904ad04fc79d4028c1882e2412 received from {username='slave'} at 127.7.39.135:36319
I20240501 14:09:04.626324 7923 tablet_copy_service.cc:433] P 47eabc135e9f4237b036097085e729d8: ending tablet copy session b4105a6d531d401faadf67b894fec79d-3137e7904ad04fc79d4028c1882e2412 on tablet 3137e7904ad04fc79d4028c1882e2412 with peer b4105a6d531d401faadf67b894fec79d
W20240501 14:09:04.638904 7708 consensus_peers.cc:485] T dcf673c75a8b48da95806e1c43309dda P 9ae5a87ca07b40a788a42734b24f66c6 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:09:04.802913 7708 consensus_peers.cc:485] T 2880045de45743bdb5358a17d93b2245 P 9ae5a87ca07b40a788a42734b24f66c6 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:09:04.820750 7839 consensus_peers.cc:485] T 14e325d8d3184c9192b8cebafb414190 P 47eabc135e9f4237b036097085e729d8 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:09:04.911994 7708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:09:04.918524 7577 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240501 14:09:04.919243 7577 consensus_peers.cc:485] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:09:05.048844 7577 consensus_peers.cc:485] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:09:05.057173 7839 consensus_peers.cc:485] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:09:05.093544 9215 raft_consensus.cc:1203] T 3137e7904ad04fc79d4028c1882e2412 P b4105a6d531d401faadf67b894fec79d [term 2 LEARNER]: Deduplicated request from leader. Original: 2.5->[2.6-2.6] Dedup: 2.6->[]
I20240501 14:09:05.557343 9982 raft_consensus.cc:1050] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8: attempting to promote NON_VOTER b4105a6d531d401faadf67b894fec79d to VOTER
I20240501 14:09:05.557752 9982 consensus_queue.cc:237] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } }
I20240501 14:09:05.559266 9215 raft_consensus.cc:1261] T 3137e7904ad04fc79d4028c1882e2412 P b4105a6d531d401faadf67b894fec79d [term 2 LEARNER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:09:05.559275 7642 raft_consensus.cc:1261] T 3137e7904ad04fc79d4028c1882e2412 P 9e44efa72e7d4421b284ae74d1cb9634 [term 2 FOLLOWER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:09:05.559870 9984 consensus_queue.cc:1035] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 14:09:05.560042 7839 consensus_peers.cc:485] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:05.560063 9941 consensus_queue.cc:1035] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:05.562389 9984 raft_consensus.cc:2932] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, b4105a6d531d401faadf67b894fec79d (127.7.39.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } }
I20240501 14:09:05.562844 9215 raft_consensus.cc:2932] T 3137e7904ad04fc79d4028c1882e2412 P b4105a6d531d401faadf67b894fec79d [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, b4105a6d531d401faadf67b894fec79d (127.7.39.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } }
I20240501 14:09:05.564188 7642 raft_consensus.cc:2932] T 3137e7904ad04fc79d4028c1882e2412 P 9e44efa72e7d4421b284ae74d1cb9634 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, b4105a6d531d401faadf67b894fec79d (127.7.39.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } }
I20240501 14:09:05.565209 7364 catalog_manager.cc:5484] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 reported cstate change: config changed from index 6 to 7, b4105a6d531d401faadf67b894fec79d (127.7.39.135) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "47eabc135e9f4237b036097085e729d8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:05.568699 7904 consensus_queue.cc:237] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } }
I20240501 14:09:05.569837 7642 raft_consensus.cc:1261] T 3137e7904ad04fc79d4028c1882e2412 P 9e44efa72e7d4421b284ae74d1cb9634 [term 2 FOLLOWER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 14:09:05.569855 9215 raft_consensus.cc:1261] T 3137e7904ad04fc79d4028c1882e2412 P b4105a6d531d401faadf67b894fec79d [term 2 FOLLOWER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 14:09:05.570223 9981 consensus_queue.cc:1035] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:05.570473 9941 consensus_queue.cc:1035] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:05.571936 9982 raft_consensus.cc:2932] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } }
I20240501 14:09:05.572425 7642 raft_consensus.cc:2932] T 3137e7904ad04fc79d4028c1882e2412 P 9e44efa72e7d4421b284ae74d1cb9634 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } }
I20240501 14:09:05.572466 9215 raft_consensus.cc:2932] T 3137e7904ad04fc79d4028c1882e2412 P b4105a6d531d401faadf67b894fec79d [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } }
I20240501 14:09:05.573436 7354 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3137e7904ad04fc79d4028c1882e2412 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:05.574164 7365 catalog_manager.cc:5484] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 reported cstate change: config changed from index 7 to 8, VOTER ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129) evicted. New cstate: current_term: 2 leader_uuid: "47eabc135e9f4237b036097085e729d8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 14:09:05.578022 7355 catalog_manager.cc:4573] TS ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 3137e7904ad04fc79d4028c1882e2412: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111)
I20240501 14:09:05.910161 7642 consensus_queue.cc:237] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:09:05.911805 7772 raft_consensus.cc:1261] T ae06bed9bd844a05ae2ac69148bf5bad P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Refusing update from remote peer 9e44efa72e7d4421b284ae74d1cb9634: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:05.912206 10005 consensus_queue.cc:1035] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:09:05.912578 7577 consensus_peers.cc:485] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:05.914074 9915 raft_consensus.cc:2932] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } }
I20240501 14:09:05.915519 7771 raft_consensus.cc:2932] T ae06bed9bd844a05ae2ac69148bf5bad P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } }
I20240501 14:09:05.916823 7364 catalog_manager.cc:5484] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "9e44efa72e7d4421b284ae74d1cb9634" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:05.919854 7642 consensus_queue.cc:237] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: NON_VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: true } }
I20240501 14:09:05.921332 7771 raft_consensus.cc:1261] T ae06bed9bd844a05ae2ac69148bf5bad P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Refusing update from remote peer 9e44efa72e7d4421b284ae74d1cb9634: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:05.921880 9915 consensus_queue.cc:1035] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 14:09:05.922472 7577 consensus_peers.cc:485] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:05.923525 9915 raft_consensus.cc:2932] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9d7c8c0f7907428eab22633702ead7ab (127.7.39.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: NON_VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: true } } }
I20240501 14:09:05.925822 7355 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ae06bed9bd844a05ae2ac69148bf5bad with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:09:05.926754 7364 catalog_manager.cc:5484] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 reported cstate change: config changed from index 4 to 5, NON_VOTER 9d7c8c0f7907428eab22633702ead7ab (127.7.39.134) added. New cstate: current_term: 1 leader_uuid: "9e44efa72e7d4421b284ae74d1cb9634" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: NON_VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:05.926818 7771 raft_consensus.cc:2932] T ae06bed9bd844a05ae2ac69148bf5bad P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9d7c8c0f7907428eab22633702ead7ab (127.7.39.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: NON_VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: true } } }
W20240501 14:09:05.928508 7578 consensus_peers.cc:485] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 -> Peer 9d7c8c0f7907428eab22633702ead7ab (127.7.39.134:38699): Couldn't send request to peer 9d7c8c0f7907428eab22633702ead7ab. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ae06bed9bd844a05ae2ac69148bf5bad. This is attempt 1: this message will repeat every 5th retry.
W20240501 14:09:06.136258 7708 consensus_peers.cc:485] T ceee52d679bf4d8e8896a6c0dde0519b P 9ae5a87ca07b40a788a42734b24f66c6 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:09:06.173465 7577 consensus_peers.cc:485] T 2cf2f85819584fdf872f6c590d11f873 P 9e44efa72e7d4421b284ae74d1cb9634 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:09:06.464123 10025 ts_tablet_manager.cc:902] T ae06bed9bd844a05ae2ac69148bf5bad P 9d7c8c0f7907428eab22633702ead7ab: Initiating tablet copy from peer 9e44efa72e7d4421b284ae74d1cb9634 (127.7.39.130:46433)
I20240501 14:09:06.464916 10025 tablet_copy_client.cc:308] T ae06bed9bd844a05ae2ac69148bf5bad P 9d7c8c0f7907428eab22633702ead7ab: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.130:46433
I20240501 14:09:06.468736 7662 tablet_copy_service.cc:140] P 9e44efa72e7d4421b284ae74d1cb9634: Received BeginTabletCopySession request for tablet ae06bed9bd844a05ae2ac69148bf5bad from peer 9d7c8c0f7907428eab22633702ead7ab ({username='slave'} at 127.7.39.134:37651)
I20240501 14:09:06.468868 7662 tablet_copy_service.cc:161] P 9e44efa72e7d4421b284ae74d1cb9634: Beginning new tablet copy session on tablet ae06bed9bd844a05ae2ac69148bf5bad from peer 9d7c8c0f7907428eab22633702ead7ab at {username='slave'} at 127.7.39.134:37651: session id = 9d7c8c0f7907428eab22633702ead7ab-ae06bed9bd844a05ae2ac69148bf5bad
I20240501 14:09:06.469975 7662 tablet_copy_source_session.cc:205] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:06.473067 10025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae06bed9bd844a05ae2ac69148bf5bad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:06.478144 10025 tablet_copy_client.cc:783] T ae06bed9bd844a05ae2ac69148bf5bad P 9d7c8c0f7907428eab22633702ead7ab: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:06.478348 10025 tablet_copy_client.cc:647] T ae06bed9bd844a05ae2ac69148bf5bad P 9d7c8c0f7907428eab22633702ead7ab: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:06.480195 10029 tablet_copy_client.cc:1002] T ae06bed9bd844a05ae2ac69148bf5bad P 9d7c8c0f7907428eab22633702ead7ab: tablet copy: Injecting latency into file download: 1500
W20240501 14:09:06.721899 7355 catalog_manager.cc:4573] TS ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 3137e7904ad04fc79d4028c1882e2412: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111)
W20240501 14:09:06.824870 7577 consensus_peers.cc:485] T df856440b07b418394477cf368b765c1 P 9e44efa72e7d4421b284ae74d1cb9634 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:09:06.947114 7839 consensus_peers.cc:485] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:09:07.014731 7839 consensus_peers.cc:485] T 14e325d8d3184c9192b8cebafb414190 P 47eabc135e9f4237b036097085e729d8 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:09:07.181077 7839 consensus_peers.cc:485] T 4d282d79e1664f7ea8912e0c5166eec3 P 47eabc135e9f4237b036097085e729d8 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:09:07.302621 7708 consensus_peers.cc:485] T dcf673c75a8b48da95806e1c43309dda P 9ae5a87ca07b40a788a42734b24f66c6 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:09:07.303094 7577 consensus_peers.cc:485] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:09:07.397039 7708 consensus_peers.cc:485] T 2880045de45743bdb5358a17d93b2245 P 9ae5a87ca07b40a788a42734b24f66c6 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:09:07.787597 7355 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111) [suppressed 15 similar messages]
W20240501 14:09:07.788633 7355 catalog_manager.cc:4573] TS ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 3137e7904ad04fc79d4028c1882e2412: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111)
I20240501 14:09:07.980670 10025 tablet_copy_client.cc:523] T ae06bed9bd844a05ae2ac69148bf5bad P 9d7c8c0f7907428eab22633702ead7ab: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:07.982650 10025 tablet_bootstrap.cc:492] T ae06bed9bd844a05ae2ac69148bf5bad P 9d7c8c0f7907428eab22633702ead7ab: Bootstrap starting.
I20240501 14:09:07.989012 10025 log.cc:824] T ae06bed9bd844a05ae2ac69148bf5bad P 9d7c8c0f7907428eab22633702ead7ab: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:08.010107 10025 tablet_bootstrap.cc:492] T ae06bed9bd844a05ae2ac69148bf5bad P 9d7c8c0f7907428eab22633702ead7ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:08.010526 10025 tablet_bootstrap.cc:492] T ae06bed9bd844a05ae2ac69148bf5bad P 9d7c8c0f7907428eab22633702ead7ab: Bootstrap complete.
I20240501 14:09:08.010850 10025 ts_tablet_manager.cc:1367] T ae06bed9bd844a05ae2ac69148bf5bad P 9d7c8c0f7907428eab22633702ead7ab: Time spent bootstrapping tablet: real 0.028s user 0.026s sys 0.000s
I20240501 14:09:08.012933 10025 raft_consensus.cc:348] T ae06bed9bd844a05ae2ac69148bf5bad P 9d7c8c0f7907428eab22633702ead7ab [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: NON_VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: true } }
I20240501 14:09:08.013186 10025 raft_consensus.cc:729] T ae06bed9bd844a05ae2ac69148bf5bad P 9d7c8c0f7907428eab22633702ead7ab [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9d7c8c0f7907428eab22633702ead7ab, State: Initialized, Role: LEARNER
I20240501 14:09:08.013583 10025 consensus_queue.cc:260] T ae06bed9bd844a05ae2ac69148bf5bad P 9d7c8c0f7907428eab22633702ead7ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: NON_VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: true } }
I20240501 14:09:08.014464 10025 ts_tablet_manager.cc:1398] T ae06bed9bd844a05ae2ac69148bf5bad P 9d7c8c0f7907428eab22633702ead7ab: Time spent starting tablet: real 0.004s user 0.007s sys 0.000s
I20240501 14:09:08.015254 7662 tablet_copy_service.cc:342] P 9e44efa72e7d4421b284ae74d1cb9634: Request end of tablet copy session 9d7c8c0f7907428eab22633702ead7ab-ae06bed9bd844a05ae2ac69148bf5bad received from {username='slave'} at 127.7.39.134:37651
I20240501 14:09:08.015367 7662 tablet_copy_service.cc:433] P 9e44efa72e7d4421b284ae74d1cb9634: ending tablet copy session 9d7c8c0f7907428eab22633702ead7ab-ae06bed9bd844a05ae2ac69148bf5bad on tablet ae06bed9bd844a05ae2ac69148bf5bad with peer 9d7c8c0f7907428eab22633702ead7ab
I20240501 14:09:08.253185 9084 raft_consensus.cc:1203] T ae06bed9bd844a05ae2ac69148bf5bad P 9d7c8c0f7907428eab22633702ead7ab [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
W20240501 14:09:08.415712 7708 consensus_peers.cc:485] T ceee52d679bf4d8e8896a6c0dde0519b P 9ae5a87ca07b40a788a42734b24f66c6 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:09:08.415771 7577 consensus_peers.cc:485] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:09:08.642475 10097 raft_consensus.cc:1050] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634: attempting to promote NON_VOTER 9d7c8c0f7907428eab22633702ead7ab to VOTER
I20240501 14:09:08.642993 10097 consensus_queue.cc:237] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } }
I20240501 14:09:08.644337 9084 raft_consensus.cc:1261] T ae06bed9bd844a05ae2ac69148bf5bad P 9d7c8c0f7907428eab22633702ead7ab [term 1 LEARNER]: Refusing update from remote peer 9e44efa72e7d4421b284ae74d1cb9634: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:08.644399 7771 raft_consensus.cc:1261] T ae06bed9bd844a05ae2ac69148bf5bad P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Refusing update from remote peer 9e44efa72e7d4421b284ae74d1cb9634: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:08.644788 10096 consensus_queue.cc:1035] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:08.644968 10099 consensus_queue.cc:1035] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 14:09:08.645176 7577 consensus_peers.cc:485] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:08.647372 10099 raft_consensus.cc:2932] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9d7c8c0f7907428eab22633702ead7ab (127.7.39.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } }
I20240501 14:09:08.647852 9084 raft_consensus.cc:2932] T ae06bed9bd844a05ae2ac69148bf5bad P 9d7c8c0f7907428eab22633702ead7ab [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9d7c8c0f7907428eab22633702ead7ab (127.7.39.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } }
I20240501 14:09:08.649250 7771 raft_consensus.cc:2932] T ae06bed9bd844a05ae2ac69148bf5bad P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9d7c8c0f7907428eab22633702ead7ab (127.7.39.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } }
I20240501 14:09:08.650120 7364 catalog_manager.cc:5484] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 reported cstate change: config changed from index 5 to 6, 9d7c8c0f7907428eab22633702ead7ab (127.7.39.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9e44efa72e7d4421b284ae74d1cb9634" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:08.653208 7642 consensus_queue.cc:237] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } }
I20240501 14:09:08.654188 9084 raft_consensus.cc:1261] T ae06bed9bd844a05ae2ac69148bf5bad P 9d7c8c0f7907428eab22633702ead7ab [term 1 FOLLOWER]: Refusing update from remote peer 9e44efa72e7d4421b284ae74d1cb9634: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:08.654512 7771 raft_consensus.cc:1261] T ae06bed9bd844a05ae2ac69148bf5bad P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Refusing update from remote peer 9e44efa72e7d4421b284ae74d1cb9634: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:08.654592 10097 consensus_queue.cc:1035] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:08.654872 10060 consensus_queue.cc:1035] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:08.656080 10097 raft_consensus.cc:2932] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } }
I20240501 14:09:08.656481 9084 raft_consensus.cc:2932] T ae06bed9bd844a05ae2ac69148bf5bad P 9d7c8c0f7907428eab22633702ead7ab [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } }
I20240501 14:09:08.656662 7771 raft_consensus.cc:2932] T ae06bed9bd844a05ae2ac69148bf5bad P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } }
I20240501 14:09:08.657567 7355 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ae06bed9bd844a05ae2ac69148bf5bad with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:08.659019 7365 catalog_manager.cc:5484] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 reported cstate change: config changed from index 6 to 7, VOTER ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129) evicted. New cstate: current_term: 1 leader_uuid: "9e44efa72e7d4421b284ae74d1cb9634" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 14:09:08.822974 7577 consensus_peers.cc:485] T 2cf2f85819584fdf872f6c590d11f873 P 9e44efa72e7d4421b284ae74d1cb9634 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:09:08.847793 7355 catalog_manager.cc:4573] TS ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet ae06bed9bd844a05ae2ac69148bf5bad: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111)
I20240501 14:09:09.029031 7642 consensus_queue.cc:237] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:09:09.030290 7904 raft_consensus.cc:1261] T ab7c1f81f7524d55bd43065890088eaf P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Refusing update from remote peer 9e44efa72e7d4421b284ae74d1cb9634: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:09.030665 10096 consensus_queue.cc:1035] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:09:09.030846 7577 consensus_peers.cc:485] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:09.032426 10099 raft_consensus.cc:2932] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } }
I20240501 14:09:09.032847 7904 raft_consensus.cc:2932] T ab7c1f81f7524d55bd43065890088eaf P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } }
I20240501 14:09:09.035143 7366 catalog_manager.cc:5484] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "9e44efa72e7d4421b284ae74d1cb9634" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:09.038353 7642 consensus_queue.cc:237] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: true } }
I20240501 14:09:09.040830 7904 raft_consensus.cc:1261] T ab7c1f81f7524d55bd43065890088eaf P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Refusing update from remote peer 9e44efa72e7d4421b284ae74d1cb9634: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:09.041350 10060 consensus_queue.cc:1035] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 14:09:09.041947 7577 consensus_peers.cc:485] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:09.043406 10099 raft_consensus.cc:2932] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9faf312806f54f2e8a4a807b234e0d1b (127.7.39.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: true } } }
I20240501 14:09:09.044620 7904 raft_consensus.cc:2932] T ab7c1f81f7524d55bd43065890088eaf P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9faf312806f54f2e8a4a807b234e0d1b (127.7.39.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: true } } }
W20240501 14:09:09.045529 7576 consensus_peers.cc:485] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 -> Peer 9faf312806f54f2e8a4a807b234e0d1b (127.7.39.133:38805): Couldn't send request to peer 9faf312806f54f2e8a4a807b234e0d1b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ab7c1f81f7524d55bd43065890088eaf. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:09.046082 7355 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ab7c1f81f7524d55bd43065890088eaf with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:09:09.046056 7366 catalog_manager.cc:5484] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 reported cstate change: config changed from index 4 to 5, NON_VOTER 9faf312806f54f2e8a4a807b234e0d1b (127.7.39.133) added. New cstate: current_term: 1 leader_uuid: "9e44efa72e7d4421b284ae74d1cb9634" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:09.289942 7577 consensus_peers.cc:485] T df856440b07b418394477cf368b765c1 P 9e44efa72e7d4421b284ae74d1cb9634 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:09:09.397511 7839 consensus_peers.cc:485] T 14e325d8d3184c9192b8cebafb414190 P 47eabc135e9f4237b036097085e729d8 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:09:09.415808 10132 ts_tablet_manager.cc:902] T ab7c1f81f7524d55bd43065890088eaf P 9faf312806f54f2e8a4a807b234e0d1b: Initiating tablet copy from peer 9e44efa72e7d4421b284ae74d1cb9634 (127.7.39.130:46433)
I20240501 14:09:09.416541 10132 tablet_copy_client.cc:308] T ab7c1f81f7524d55bd43065890088eaf P 9faf312806f54f2e8a4a807b234e0d1b: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.130:46433
I20240501 14:09:09.419672 7662 tablet_copy_service.cc:140] P 9e44efa72e7d4421b284ae74d1cb9634: Received BeginTabletCopySession request for tablet ab7c1f81f7524d55bd43065890088eaf from peer 9faf312806f54f2e8a4a807b234e0d1b ({username='slave'} at 127.7.39.133:34271)
I20240501 14:09:09.419798 7662 tablet_copy_service.cc:161] P 9e44efa72e7d4421b284ae74d1cb9634: Beginning new tablet copy session on tablet ab7c1f81f7524d55bd43065890088eaf from peer 9faf312806f54f2e8a4a807b234e0d1b at {username='slave'} at 127.7.39.133:34271: session id = 9faf312806f54f2e8a4a807b234e0d1b-ab7c1f81f7524d55bd43065890088eaf
I20240501 14:09:09.420533 7662 tablet_copy_source_session.cc:205] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:09.423373 10132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab7c1f81f7524d55bd43065890088eaf. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 14:09:09.427443 7839 consensus_peers.cc:485] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:09:09.428934 10132 tablet_copy_client.cc:783] T ab7c1f81f7524d55bd43065890088eaf P 9faf312806f54f2e8a4a807b234e0d1b: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:09.429145 10132 tablet_copy_client.cc:647] T ab7c1f81f7524d55bd43065890088eaf P 9faf312806f54f2e8a4a807b234e0d1b: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:09.430267 10136 tablet_copy_client.cc:1002] T ab7c1f81f7524d55bd43065890088eaf P 9faf312806f54f2e8a4a807b234e0d1b: tablet copy: Injecting latency into file download: 1500
W20240501 14:09:09.709164 7839 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240501 14:09:09.709797 7839 consensus_peers.cc:485] T 4d282d79e1664f7ea8912e0c5166eec3 P 47eabc135e9f4237b036097085e729d8 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:09:09.865056 7355 catalog_manager.cc:4573] TS ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 3137e7904ad04fc79d4028c1882e2412: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111)
W20240501 14:09:09.871266 7708 consensus_peers.cc:485] T dcf673c75a8b48da95806e1c43309dda P 9ae5a87ca07b40a788a42734b24f66c6 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:09:09.944342 7577 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240501 14:09:10.245592 7708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:09:10.246209 7708 consensus_peers.cc:485] T 2880045de45743bdb5358a17d93b2245 P 9ae5a87ca07b40a788a42734b24f66c6 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:09:10.818420 7708 consensus_peers.cc:485] T ceee52d679bf4d8e8896a6c0dde0519b P 9ae5a87ca07b40a788a42734b24f66c6 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:09:10.871155 7355 catalog_manager.cc:4573] TS ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet ae06bed9bd844a05ae2ac69148bf5bad: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111)
I20240501 14:09:10.930737 10132 tablet_copy_client.cc:523] T ab7c1f81f7524d55bd43065890088eaf P 9faf312806f54f2e8a4a807b234e0d1b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:10.932770 10132 tablet_bootstrap.cc:492] T ab7c1f81f7524d55bd43065890088eaf P 9faf312806f54f2e8a4a807b234e0d1b: Bootstrap starting.
I20240501 14:09:10.938313 10132 log.cc:824] T ab7c1f81f7524d55bd43065890088eaf P 9faf312806f54f2e8a4a807b234e0d1b: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:10.957902 10132 tablet_bootstrap.cc:492] T ab7c1f81f7524d55bd43065890088eaf P 9faf312806f54f2e8a4a807b234e0d1b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:10.958278 10132 tablet_bootstrap.cc:492] T ab7c1f81f7524d55bd43065890088eaf P 9faf312806f54f2e8a4a807b234e0d1b: Bootstrap complete.
I20240501 14:09:10.958595 10132 ts_tablet_manager.cc:1367] T ab7c1f81f7524d55bd43065890088eaf P 9faf312806f54f2e8a4a807b234e0d1b: Time spent bootstrapping tablet: real 0.026s user 0.019s sys 0.004s
I20240501 14:09:10.960547 10132 raft_consensus.cc:348] T ab7c1f81f7524d55bd43065890088eaf P 9faf312806f54f2e8a4a807b234e0d1b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: true } }
I20240501 14:09:10.960754 10132 raft_consensus.cc:729] T ab7c1f81f7524d55bd43065890088eaf P 9faf312806f54f2e8a4a807b234e0d1b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9faf312806f54f2e8a4a807b234e0d1b, State: Initialized, Role: LEARNER
I20240501 14:09:10.961050 10132 consensus_queue.cc:260] T ab7c1f81f7524d55bd43065890088eaf P 9faf312806f54f2e8a4a807b234e0d1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: true } }
I20240501 14:09:10.961807 10132 ts_tablet_manager.cc:1398] T ab7c1f81f7524d55bd43065890088eaf P 9faf312806f54f2e8a4a807b234e0d1b: Time spent starting tablet: real 0.003s user 0.006s sys 0.000s
I20240501 14:09:10.962535 7662 tablet_copy_service.cc:342] P 9e44efa72e7d4421b284ae74d1cb9634: Request end of tablet copy session 9faf312806f54f2e8a4a807b234e0d1b-ab7c1f81f7524d55bd43065890088eaf received from {username='slave'} at 127.7.39.133:34271
I20240501 14:09:10.962610 7662 tablet_copy_service.cc:433] P 9e44efa72e7d4421b284ae74d1cb9634: ending tablet copy session 9faf312806f54f2e8a4a807b234e0d1b-ab7c1f81f7524d55bd43065890088eaf on tablet ab7c1f81f7524d55bd43065890088eaf with peer 9faf312806f54f2e8a4a807b234e0d1b
I20240501 14:09:10.988373 8953 raft_consensus.cc:1203] T ab7c1f81f7524d55bd43065890088eaf P 9faf312806f54f2e8a4a807b234e0d1b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
W20240501 14:09:11.335492 7577 consensus_peers.cc:485] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:09:11.440353 7577 consensus_peers.cc:485] T 2cf2f85819584fdf872f6c590d11f873 P 9e44efa72e7d4421b284ae74d1cb9634 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:09:11.489722 10156 raft_consensus.cc:1050] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634: attempting to promote NON_VOTER 9faf312806f54f2e8a4a807b234e0d1b to VOTER
I20240501 14:09:11.490124 10156 consensus_queue.cc:237] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } }
I20240501 14:09:11.491586 7904 raft_consensus.cc:1261] T ab7c1f81f7524d55bd43065890088eaf P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Refusing update from remote peer 9e44efa72e7d4421b284ae74d1cb9634: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:11.491633 8953 raft_consensus.cc:1261] T ab7c1f81f7524d55bd43065890088eaf P 9faf312806f54f2e8a4a807b234e0d1b [term 1 LEARNER]: Refusing update from remote peer 9e44efa72e7d4421b284ae74d1cb9634: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:11.492036 10196 consensus_queue.cc:1035] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:11.492194 10199 consensus_queue.cc:1035] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 14:09:11.492316 7577 consensus_peers.cc:485] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:11.494684 10199 raft_consensus.cc:2932] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9faf312806f54f2e8a4a807b234e0d1b (127.7.39.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } }
I20240501 14:09:11.495240 8953 raft_consensus.cc:2932] T ab7c1f81f7524d55bd43065890088eaf P 9faf312806f54f2e8a4a807b234e0d1b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9faf312806f54f2e8a4a807b234e0d1b (127.7.39.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } }
I20240501 14:09:11.496649 7904 raft_consensus.cc:2932] T ab7c1f81f7524d55bd43065890088eaf P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9faf312806f54f2e8a4a807b234e0d1b (127.7.39.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } }
I20240501 14:09:11.498044 7365 catalog_manager.cc:5484] T ab7c1f81f7524d55bd43065890088eaf P 9faf312806f54f2e8a4a807b234e0d1b reported cstate change: config changed from index 5 to 6, 9faf312806f54f2e8a4a807b234e0d1b (127.7.39.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9e44efa72e7d4421b284ae74d1cb9634" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } }
I20240501 14:09:11.501515 7642 consensus_queue.cc:237] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } }
I20240501 14:09:11.502559 8953 raft_consensus.cc:1261] T ab7c1f81f7524d55bd43065890088eaf P 9faf312806f54f2e8a4a807b234e0d1b [term 1 FOLLOWER]: Refusing update from remote peer 9e44efa72e7d4421b284ae74d1cb9634: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:11.502625 7904 raft_consensus.cc:1261] T ab7c1f81f7524d55bd43065890088eaf P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Refusing update from remote peer 9e44efa72e7d4421b284ae74d1cb9634: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:11.503005 10198 consensus_queue.cc:1035] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:11.503201 10199 consensus_queue.cc:1035] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:11.504606 10196 raft_consensus.cc:2932] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } }
I20240501 14:09:11.505012 8953 raft_consensus.cc:2932] T ab7c1f81f7524d55bd43065890088eaf P 9faf312806f54f2e8a4a807b234e0d1b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } }
I20240501 14:09:11.505064 7904 raft_consensus.cc:2932] T ab7c1f81f7524d55bd43065890088eaf P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } }
I20240501 14:09:11.506140 7355 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ab7c1f81f7524d55bd43065890088eaf with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:11.507287 7364 catalog_manager.cc:5484] T ab7c1f81f7524d55bd43065890088eaf P 9faf312806f54f2e8a4a807b234e0d1b reported cstate change: config changed from index 6 to 7, VOTER ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129) evicted. New cstate: current_term: 1 leader_uuid: "9e44efa72e7d4421b284ae74d1cb9634" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } }
W20240501 14:09:11.671149 7577 consensus_peers.cc:485] T df856440b07b418394477cf368b765c1 P 9e44efa72e7d4421b284ae74d1cb9634 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:09:11.734575 7904 consensus_queue.cc:237] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:09:11.736449 7771 raft_consensus.cc:1261] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:11.737296 10080 consensus_queue.cc:1035] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:09:11.737896 7839 consensus_peers.cc:485] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:11.740567 10080 raft_consensus.cc:2932] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } }
I20240501 14:09:11.741137 7771 raft_consensus.cc:2932] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } }
I20240501 14:09:11.743584 7366 catalog_manager.cc:5484] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "47eabc135e9f4237b036097085e729d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:11.747627 7904 consensus_queue.cc:237] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: NON_VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: true } }
W20240501 14:09:11.749565 7839 consensus_peers.cc:485] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 -> Peer b4105a6d531d401faadf67b894fec79d (127.7.39.135:39611): Couldn't send request to peer b4105a6d531d401faadf67b894fec79d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2112bf96dc3343678aa6f46bf2a3b1f2. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:11.750344 7771 raft_consensus.cc:1261] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 14:09:11.750720 7839 consensus_peers.cc:485] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:11.750969 10223 consensus_queue.cc:1035] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:11.752526 10080 raft_consensus.cc:2932] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b4105a6d531d401faadf67b894fec79d (127.7.39.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: NON_VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: true } } }
I20240501 14:09:11.752956 7771 raft_consensus.cc:2932] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b4105a6d531d401faadf67b894fec79d (127.7.39.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: NON_VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: true } } }
I20240501 14:09:11.753857 7354 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2112bf96dc3343678aa6f46bf2a3b1f2 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:09:11.754772 7365 catalog_manager.cc:5484] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 reported cstate change: config changed from index 4 to 5, NON_VOTER b4105a6d531d401faadf67b894fec79d (127.7.39.135) added. New cstate: current_term: 1 leader_uuid: "47eabc135e9f4237b036097085e729d8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: NON_VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:12.071478 7839 consensus_peers.cc:485] T 14e325d8d3184c9192b8cebafb414190 P 47eabc135e9f4237b036097085e729d8 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:09:12.119864 7355 catalog_manager.cc:4573] TS ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet ab7c1f81f7524d55bd43065890088eaf: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111)
I20240501 14:09:12.131000 10234 ts_tablet_manager.cc:902] T 2112bf96dc3343678aa6f46bf2a3b1f2 P b4105a6d531d401faadf67b894fec79d: Initiating tablet copy from peer 47eabc135e9f4237b036097085e729d8 (127.7.39.132:42369)
I20240501 14:09:12.131750 10234 tablet_copy_client.cc:308] T 2112bf96dc3343678aa6f46bf2a3b1f2 P b4105a6d531d401faadf67b894fec79d: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.132:42369
I20240501 14:09:12.132643 7923 tablet_copy_service.cc:140] P 47eabc135e9f4237b036097085e729d8: Received BeginTabletCopySession request for tablet 2112bf96dc3343678aa6f46bf2a3b1f2 from peer b4105a6d531d401faadf67b894fec79d ({username='slave'} at 127.7.39.135:36319)
I20240501 14:09:12.132781 7923 tablet_copy_service.cc:161] P 47eabc135e9f4237b036097085e729d8: Beginning new tablet copy session on tablet 2112bf96dc3343678aa6f46bf2a3b1f2 from peer b4105a6d531d401faadf67b894fec79d at {username='slave'} at 127.7.39.135:36319: session id = b4105a6d531d401faadf67b894fec79d-2112bf96dc3343678aa6f46bf2a3b1f2
I20240501 14:09:12.134174 7923 tablet_copy_source_session.cc:205] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:12.135687 10234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2112bf96dc3343678aa6f46bf2a3b1f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:12.138070 10234 tablet_copy_client.cc:783] T 2112bf96dc3343678aa6f46bf2a3b1f2 P b4105a6d531d401faadf67b894fec79d: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:12.138291 10234 tablet_copy_client.cc:647] T 2112bf96dc3343678aa6f46bf2a3b1f2 P b4105a6d531d401faadf67b894fec79d: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:12.139801 10236 tablet_copy_client.cc:1002] T 2112bf96dc3343678aa6f46bf2a3b1f2 P b4105a6d531d401faadf67b894fec79d: tablet copy: Injecting latency into file download: 1500
W20240501 14:09:12.374030 7708 consensus_peers.cc:485] T dcf673c75a8b48da95806e1c43309dda P 9ae5a87ca07b40a788a42734b24f66c6 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:09:12.386202 7839 consensus_peers.cc:485] T 4d282d79e1664f7ea8912e0c5166eec3 P 47eabc135e9f4237b036097085e729d8 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:09:12.847463 7708 consensus_peers.cc:485] T 2880045de45743bdb5358a17d93b2245 P 9ae5a87ca07b40a788a42734b24f66c6 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:09:12.912793 7708 consensus_peers.cc:485] T ceee52d679bf4d8e8896a6c0dde0519b P 9ae5a87ca07b40a788a42734b24f66c6 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:09:12.944602 7355 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111) [suppressed 17 similar messages]
I20240501 14:09:13.640352 10234 tablet_copy_client.cc:523] T 2112bf96dc3343678aa6f46bf2a3b1f2 P b4105a6d531d401faadf67b894fec79d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:13.643685 10234 tablet_bootstrap.cc:492] T 2112bf96dc3343678aa6f46bf2a3b1f2 P b4105a6d531d401faadf67b894fec79d: Bootstrap starting.
W20240501 14:09:13.689947 7355 catalog_manager.cc:4573] TS ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet ab7c1f81f7524d55bd43065890088eaf: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111)
I20240501 14:09:13.701357 10234 tablet_bootstrap.cc:492] T 2112bf96dc3343678aa6f46bf2a3b1f2 P b4105a6d531d401faadf67b894fec79d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:13.702153 10234 tablet_bootstrap.cc:492] T 2112bf96dc3343678aa6f46bf2a3b1f2 P b4105a6d531d401faadf67b894fec79d: Bootstrap complete.
I20240501 14:09:13.702780 10234 ts_tablet_manager.cc:1367] T 2112bf96dc3343678aa6f46bf2a3b1f2 P b4105a6d531d401faadf67b894fec79d: Time spent bootstrapping tablet: real 0.059s user 0.055s sys 0.000s
I20240501 14:09:13.704365 10234 raft_consensus.cc:348] T 2112bf96dc3343678aa6f46bf2a3b1f2 P b4105a6d531d401faadf67b894fec79d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: NON_VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: true } }
I20240501 14:09:13.704661 10234 raft_consensus.cc:729] T 2112bf96dc3343678aa6f46bf2a3b1f2 P b4105a6d531d401faadf67b894fec79d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b4105a6d531d401faadf67b894fec79d, State: Initialized, Role: LEARNER
I20240501 14:09:13.704972 10234 consensus_queue.cc:260] T 2112bf96dc3343678aa6f46bf2a3b1f2 P b4105a6d531d401faadf67b894fec79d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: NON_VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: true } }
I20240501 14:09:13.705861 10234 ts_tablet_manager.cc:1398] T 2112bf96dc3343678aa6f46bf2a3b1f2 P b4105a6d531d401faadf67b894fec79d: Time spent starting tablet: real 0.003s user 0.006s sys 0.000s
I20240501 14:09:13.707099 7923 tablet_copy_service.cc:342] P 47eabc135e9f4237b036097085e729d8: Request end of tablet copy session b4105a6d531d401faadf67b894fec79d-2112bf96dc3343678aa6f46bf2a3b1f2 received from {username='slave'} at 127.7.39.135:36319
I20240501 14:09:13.707290 7923 tablet_copy_service.cc:433] P 47eabc135e9f4237b036097085e729d8: ending tablet copy session b4105a6d531d401faadf67b894fec79d-2112bf96dc3343678aa6f46bf2a3b1f2 on tablet 2112bf96dc3343678aa6f46bf2a3b1f2 with peer b4105a6d531d401faadf67b894fec79d
I20240501 14:09:13.814817 9215 raft_consensus.cc:1203] T 2112bf96dc3343678aa6f46bf2a3b1f2 P b4105a6d531d401faadf67b894fec79d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
W20240501 14:09:14.025640 7577 consensus_peers.cc:485] T 2cf2f85819584fdf872f6c590d11f873 P 9e44efa72e7d4421b284ae74d1cb9634 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:09:14.256085 7577 consensus_peers.cc:485] T df856440b07b418394477cf368b765c1 P 9e44efa72e7d4421b284ae74d1cb9634 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:09:14.400463 10304 raft_consensus.cc:1050] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8: attempting to promote NON_VOTER b4105a6d531d401faadf67b894fec79d to VOTER
I20240501 14:09:14.400964 10304 consensus_queue.cc:237] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } }
I20240501 14:09:14.402319 9215 raft_consensus.cc:1261] T 2112bf96dc3343678aa6f46bf2a3b1f2 P b4105a6d531d401faadf67b894fec79d [term 1 LEARNER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:14.402477 7771 raft_consensus.cc:1261] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:14.402777 10306 consensus_queue.cc:1035] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:14.402966 10307 consensus_queue.cc:1035] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 14:09:14.403347 7839 consensus_peers.cc:485] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:14.404875 10306 raft_consensus.cc:2932] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b4105a6d531d401faadf67b894fec79d (127.7.39.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } }
I20240501 14:09:14.405331 9215 raft_consensus.cc:2932] T 2112bf96dc3343678aa6f46bf2a3b1f2 P b4105a6d531d401faadf67b894fec79d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b4105a6d531d401faadf67b894fec79d (127.7.39.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } }
I20240501 14:09:14.407243 7771 raft_consensus.cc:2932] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b4105a6d531d401faadf67b894fec79d (127.7.39.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } }
I20240501 14:09:14.407534 7365 catalog_manager.cc:5484] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 reported cstate change: config changed from index 5 to 6, b4105a6d531d401faadf67b894fec79d (127.7.39.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "47eabc135e9f4237b036097085e729d8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:14.412338 7903 consensus_queue.cc:237] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } }
I20240501 14:09:14.413516 9215 raft_consensus.cc:1261] T 2112bf96dc3343678aa6f46bf2a3b1f2 P b4105a6d531d401faadf67b894fec79d [term 1 FOLLOWER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:14.413554 7771 raft_consensus.cc:1261] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:14.413923 10307 consensus_queue.cc:1035] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:14.414068 10304 consensus_queue.cc:1035] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:14.415655 10304 raft_consensus.cc:2932] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } }
I20240501 14:09:14.416210 9215 raft_consensus.cc:2932] T 2112bf96dc3343678aa6f46bf2a3b1f2 P b4105a6d531d401faadf67b894fec79d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } }
I20240501 14:09:14.416210 7771 raft_consensus.cc:2932] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } }
I20240501 14:09:14.417865 7354 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2112bf96dc3343678aa6f46bf2a3b1f2 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:14.418728 7364 catalog_manager.cc:5484] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9ae5a87ca07b40a788a42734b24f66c6 reported cstate change: config changed from index 6 to 7, VOTER ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129) evicted. New cstate: current_term: 1 leader_uuid: "47eabc135e9f4237b036097085e729d8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } }
W20240501 14:09:14.573482 7839 consensus_peers.cc:485] T 14e325d8d3184c9192b8cebafb414190 P 47eabc135e9f4237b036097085e729d8 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:09:14.773020 7708 consensus_peers.cc:485] T dcf673c75a8b48da95806e1c43309dda P 9ae5a87ca07b40a788a42734b24f66c6 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:09:14.773989 7355 catalog_manager.cc:4573] TS ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 2112bf96dc3343678aa6f46bf2a3b1f2: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111)
I20240501 14:09:14.793639 7882 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:14.793738 9064 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:14.802310 9326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:14.802500 9195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:14.803176 7620 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:14.824080 7752 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:14.827601 8931 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:14.915817 9326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:14.919553 7752 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:14.921247 8931 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:14.921665 9195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:14.924085 9064 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:14.928442 7882 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:14.934798 7620 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:09:14.956257 7839 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 14:09:14.981967 7641 consensus_queue.cc:237] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: true } }
I20240501 14:09:14.983670 7771 raft_consensus.cc:1261] T ae06bed9bd844a05ae2ac69148bf5bad P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Refusing update from remote peer 9e44efa72e7d4421b284ae74d1cb9634: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:14.983732 9084 raft_consensus.cc:1261] T ae06bed9bd844a05ae2ac69148bf5bad P 9d7c8c0f7907428eab22633702ead7ab [term 1 FOLLOWER]: Refusing update from remote peer 9e44efa72e7d4421b284ae74d1cb9634: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240501 14:09:14.983848 7576 consensus_peers.cc:485] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 -> Peer 9faf312806f54f2e8a4a807b234e0d1b (127.7.39.133:38805): Couldn't send request to peer 9faf312806f54f2e8a4a807b234e0d1b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ae06bed9bd844a05ae2ac69148bf5bad. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:14.984256 10404 consensus_queue.cc:1035] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:14.984488 10268 consensus_queue.cc:1035] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:14.986372 10397 raft_consensus.cc:2932] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9faf312806f54f2e8a4a807b234e0d1b (127.7.39.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: true } } }
I20240501 14:09:14.986972 7771 raft_consensus.cc:2932] T ae06bed9bd844a05ae2ac69148bf5bad P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9faf312806f54f2e8a4a807b234e0d1b (127.7.39.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: true } } }
I20240501 14:09:14.987191 9084 raft_consensus.cc:2932] T ae06bed9bd844a05ae2ac69148bf5bad P 9d7c8c0f7907428eab22633702ead7ab [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9faf312806f54f2e8a4a807b234e0d1b (127.7.39.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: true } } }
I20240501 14:09:14.990412 7366 catalog_manager.cc:5484] T ae06bed9bd844a05ae2ac69148bf5bad P 9ae5a87ca07b40a788a42734b24f66c6 reported cstate change: config changed from index 7 to 8, NON_VOTER 9faf312806f54f2e8a4a807b234e0d1b (127.7.39.133) added. New cstate: current_term: 1 leader_uuid: "9e44efa72e7d4421b284ae74d1cb9634" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: true } } }
I20240501 14:09:14.998880 7903 consensus_queue.cc:237] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: NON_VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: true } }
I20240501 14:09:15.000710 9346 raft_consensus.cc:1261] T c747bd6504c540cebded0019d82495b4 P 9d32e843754e44d0b9aae6f472a4d9a2 [term 1 FOLLOWER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:15.000857 7771 raft_consensus.cc:1261] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:15.001243 10307 consensus_queue.cc:1035] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:15.001454 10418 consensus_queue.cc:1035] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:15.003613 10418 raft_consensus.cc:2932] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9d7c8c0f7907428eab22633702ead7ab (127.7.39.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: NON_VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: true } } }
W20240501 14:09:15.004247 7839 consensus_peers.cc:485] T 4d282d79e1664f7ea8912e0c5166eec3 P 47eabc135e9f4237b036097085e729d8 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:09:15.004756 7840 consensus_peers.cc:485] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 -> Peer 9d7c8c0f7907428eab22633702ead7ab (127.7.39.134:38699): Couldn't send request to peer 9d7c8c0f7907428eab22633702ead7ab. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c747bd6504c540cebded0019d82495b4. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:15.004863 7771 raft_consensus.cc:2932] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9d7c8c0f7907428eab22633702ead7ab (127.7.39.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: NON_VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: true } } }
I20240501 14:09:15.007437 9346 raft_consensus.cc:2932] T c747bd6504c540cebded0019d82495b4 P 9d32e843754e44d0b9aae6f472a4d9a2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9d7c8c0f7907428eab22633702ead7ab (127.7.39.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: NON_VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: true } } }
I20240501 14:09:15.007535 7364 catalog_manager.cc:5484] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 reported cstate change: config changed from index 7 to 8, NON_VOTER 9d7c8c0f7907428eab22633702ead7ab (127.7.39.134) added. New cstate: current_term: 1 leader_uuid: "47eabc135e9f4237b036097085e729d8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: NON_VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: true } } }
I20240501 14:09:15.014003 7903 consensus_queue.cc:237] T 0bfcc65d57eb4c3597a761f878a09da2 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: NON_VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: true } }
W20240501 14:09:15.015141 7577 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111) [suppressed 23 similar messages]
I20240501 14:09:15.015630 7641 raft_consensus.cc:1261] T 0bfcc65d57eb4c3597a761f878a09da2 P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 FOLLOWER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:15.015988 7771 raft_consensus.cc:1261] T 0bfcc65d57eb4c3597a761f878a09da2 P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:15.015998 10418 consensus_queue.cc:1035] T 0bfcc65d57eb4c3597a761f878a09da2 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:09:15.016201 7839 consensus_peers.cc:485] T 0bfcc65d57eb4c3597a761f878a09da2 P 47eabc135e9f4237b036097085e729d8 -> Peer b4105a6d531d401faadf67b894fec79d (127.7.39.135:39611): Couldn't send request to peer b4105a6d531d401faadf67b894fec79d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0bfcc65d57eb4c3597a761f878a09da2. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:15.016386 10418 consensus_queue.cc:1035] T 0bfcc65d57eb4c3597a761f878a09da2 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:15.017788 10307 raft_consensus.cc:2932] T 0bfcc65d57eb4c3597a761f878a09da2 P 47eabc135e9f4237b036097085e729d8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b4105a6d531d401faadf67b894fec79d (127.7.39.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: NON_VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: true } } }
I20240501 14:09:15.018218 7641 raft_consensus.cc:2932] T 0bfcc65d57eb4c3597a761f878a09da2 P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b4105a6d531d401faadf67b894fec79d (127.7.39.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: NON_VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: true } } }
I20240501 14:09:15.018585 7771 raft_consensus.cc:2932] T 0bfcc65d57eb4c3597a761f878a09da2 P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b4105a6d531d401faadf67b894fec79d (127.7.39.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: NON_VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: true } } }
I20240501 14:09:15.020848 7366 catalog_manager.cc:5484] T 0bfcc65d57eb4c3597a761f878a09da2 P 9ae5a87ca07b40a788a42734b24f66c6 reported cstate change: config changed from index -1 to 4, NON_VOTER b4105a6d531d401faadf67b894fec79d (127.7.39.135) added. New cstate: current_term: 1 leader_uuid: "47eabc135e9f4237b036097085e729d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: NON_VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: true } } }
W20240501 14:09:15.358371 7708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:09:15.405197 7708 consensus_peers.cc:485] T ceee52d679bf4d8e8896a6c0dde0519b P 9ae5a87ca07b40a788a42734b24f66c6 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:09:15.444866 7708 consensus_peers.cc:485] T 2880045de45743bdb5358a17d93b2245 P 9ae5a87ca07b40a788a42734b24f66c6 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:09:15.494277 10476 ts_tablet_manager.cc:902] T 0bfcc65d57eb4c3597a761f878a09da2 P b4105a6d531d401faadf67b894fec79d: Initiating tablet copy from peer 47eabc135e9f4237b036097085e729d8 (127.7.39.132:42369)
I20240501 14:09:15.494772 10476 tablet_copy_client.cc:308] T 0bfcc65d57eb4c3597a761f878a09da2 P b4105a6d531d401faadf67b894fec79d: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.132:42369
I20240501 14:09:15.495378 7923 tablet_copy_service.cc:140] P 47eabc135e9f4237b036097085e729d8: Received BeginTabletCopySession request for tablet 0bfcc65d57eb4c3597a761f878a09da2 from peer b4105a6d531d401faadf67b894fec79d ({username='slave'} at 127.7.39.135:36319)
I20240501 14:09:15.495501 7923 tablet_copy_service.cc:161] P 47eabc135e9f4237b036097085e729d8: Beginning new tablet copy session on tablet 0bfcc65d57eb4c3597a761f878a09da2 from peer b4105a6d531d401faadf67b894fec79d at {username='slave'} at 127.7.39.135:36319: session id = b4105a6d531d401faadf67b894fec79d-0bfcc65d57eb4c3597a761f878a09da2
I20240501 14:09:15.496287 7923 tablet_copy_source_session.cc:205] T 0bfcc65d57eb4c3597a761f878a09da2 P 47eabc135e9f4237b036097085e729d8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:15.497164 10476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bfcc65d57eb4c3597a761f878a09da2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:15.499076 10476 tablet_copy_client.cc:783] T 0bfcc65d57eb4c3597a761f878a09da2 P b4105a6d531d401faadf67b894fec79d: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:15.499243 10476 tablet_copy_client.cc:647] T 0bfcc65d57eb4c3597a761f878a09da2 P b4105a6d531d401faadf67b894fec79d: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:15.500226 10480 tablet_copy_client.cc:1002] T 0bfcc65d57eb4c3597a761f878a09da2 P b4105a6d531d401faadf67b894fec79d: tablet copy: Injecting latency into file download: 1500
I20240501 14:09:15.540503 10481 ts_tablet_manager.cc:902] T ae06bed9bd844a05ae2ac69148bf5bad P 9faf312806f54f2e8a4a807b234e0d1b: Initiating tablet copy from peer 9e44efa72e7d4421b284ae74d1cb9634 (127.7.39.130:46433)
I20240501 14:09:15.541015 10481 tablet_copy_client.cc:308] T ae06bed9bd844a05ae2ac69148bf5bad P 9faf312806f54f2e8a4a807b234e0d1b: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.130:46433
I20240501 14:09:15.541563 7662 tablet_copy_service.cc:140] P 9e44efa72e7d4421b284ae74d1cb9634: Received BeginTabletCopySession request for tablet ae06bed9bd844a05ae2ac69148bf5bad from peer 9faf312806f54f2e8a4a807b234e0d1b ({username='slave'} at 127.7.39.133:34271)
I20240501 14:09:15.541668 7662 tablet_copy_service.cc:161] P 9e44efa72e7d4421b284ae74d1cb9634: Beginning new tablet copy session on tablet ae06bed9bd844a05ae2ac69148bf5bad from peer 9faf312806f54f2e8a4a807b234e0d1b at {username='slave'} at 127.7.39.133:34271: session id = 9faf312806f54f2e8a4a807b234e0d1b-ae06bed9bd844a05ae2ac69148bf5bad
I20240501 14:09:15.542519 7662 tablet_copy_source_session.cc:205] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:15.543385 10481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae06bed9bd844a05ae2ac69148bf5bad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:15.545289 10481 tablet_copy_client.cc:783] T ae06bed9bd844a05ae2ac69148bf5bad P 9faf312806f54f2e8a4a807b234e0d1b: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:15.545449 10481 tablet_copy_client.cc:647] T ae06bed9bd844a05ae2ac69148bf5bad P 9faf312806f54f2e8a4a807b234e0d1b: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:15.546528 10485 tablet_copy_client.cc:1002] T ae06bed9bd844a05ae2ac69148bf5bad P 9faf312806f54f2e8a4a807b234e0d1b: tablet copy: Injecting latency into file download: 1500
I20240501 14:09:15.617736 10489 ts_tablet_manager.cc:902] T c747bd6504c540cebded0019d82495b4 P 9d7c8c0f7907428eab22633702ead7ab: Initiating tablet copy from peer 47eabc135e9f4237b036097085e729d8 (127.7.39.132:42369)
I20240501 14:09:15.618242 10489 tablet_copy_client.cc:308] T c747bd6504c540cebded0019d82495b4 P 9d7c8c0f7907428eab22633702ead7ab: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.132:42369
I20240501 14:09:15.621584 7923 tablet_copy_service.cc:140] P 47eabc135e9f4237b036097085e729d8: Received BeginTabletCopySession request for tablet c747bd6504c540cebded0019d82495b4 from peer 9d7c8c0f7907428eab22633702ead7ab ({username='slave'} at 127.7.39.134:48187)
I20240501 14:09:15.621690 7923 tablet_copy_service.cc:161] P 47eabc135e9f4237b036097085e729d8: Beginning new tablet copy session on tablet c747bd6504c540cebded0019d82495b4 from peer 9d7c8c0f7907428eab22633702ead7ab at {username='slave'} at 127.7.39.134:48187: session id = 9d7c8c0f7907428eab22633702ead7ab-c747bd6504c540cebded0019d82495b4
I20240501 14:09:15.622471 7923 tablet_copy_source_session.cc:205] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:15.623322 10489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c747bd6504c540cebded0019d82495b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:15.625236 10489 tablet_copy_client.cc:783] T c747bd6504c540cebded0019d82495b4 P 9d7c8c0f7907428eab22633702ead7ab: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:15.625389 10489 tablet_copy_client.cc:647] T c747bd6504c540cebded0019d82495b4 P 9d7c8c0f7907428eab22633702ead7ab: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:15.626374 10493 tablet_copy_client.cc:1002] T c747bd6504c540cebded0019d82495b4 P 9d7c8c0f7907428eab22633702ead7ab: tablet copy: Injecting latency into file download: 1500
W20240501 14:09:15.782409 7355 catalog_manager.cc:4573] TS ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet ab7c1f81f7524d55bd43065890088eaf: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111)
W20240501 14:09:16.554556 7577 consensus_peers.cc:485] T 2cf2f85819584fdf872f6c590d11f873 P 9e44efa72e7d4421b284ae74d1cb9634 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:09:16.588925 7577 consensus_peers.cc:485] T df856440b07b418394477cf368b765c1 P 9e44efa72e7d4421b284ae74d1cb9634 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:09:17.000500 10476 tablet_copy_client.cc:523] T 0bfcc65d57eb4c3597a761f878a09da2 P b4105a6d531d401faadf67b894fec79d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:17.001897 10476 tablet_bootstrap.cc:492] T 0bfcc65d57eb4c3597a761f878a09da2 P b4105a6d531d401faadf67b894fec79d: Bootstrap starting.
I20240501 14:09:17.019284 10476 tablet_bootstrap.cc:492] T 0bfcc65d57eb4c3597a761f878a09da2 P b4105a6d531d401faadf67b894fec79d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:17.019569 10476 tablet_bootstrap.cc:492] T 0bfcc65d57eb4c3597a761f878a09da2 P b4105a6d531d401faadf67b894fec79d: Bootstrap complete.
I20240501 14:09:17.019742 10476 ts_tablet_manager.cc:1367] T 0bfcc65d57eb4c3597a761f878a09da2 P b4105a6d531d401faadf67b894fec79d: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.004s
I20240501 14:09:17.020176 10476 raft_consensus.cc:348] T 0bfcc65d57eb4c3597a761f878a09da2 P b4105a6d531d401faadf67b894fec79d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: NON_VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: true } }
I20240501 14:09:17.020290 10476 raft_consensus.cc:729] T 0bfcc65d57eb4c3597a761f878a09da2 P b4105a6d531d401faadf67b894fec79d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b4105a6d531d401faadf67b894fec79d, State: Initialized, Role: LEARNER
I20240501 14:09:17.020396 10476 consensus_queue.cc:260] T 0bfcc65d57eb4c3597a761f878a09da2 P b4105a6d531d401faadf67b894fec79d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: NON_VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: true } }
I20240501 14:09:17.020718 10476 ts_tablet_manager.cc:1398] T 0bfcc65d57eb4c3597a761f878a09da2 P b4105a6d531d401faadf67b894fec79d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:17.021342 7923 tablet_copy_service.cc:342] P 47eabc135e9f4237b036097085e729d8: Request end of tablet copy session b4105a6d531d401faadf67b894fec79d-0bfcc65d57eb4c3597a761f878a09da2 received from {username='slave'} at 127.7.39.135:36319
I20240501 14:09:17.021440 7923 tablet_copy_service.cc:433] P 47eabc135e9f4237b036097085e729d8: ending tablet copy session b4105a6d531d401faadf67b894fec79d-0bfcc65d57eb4c3597a761f878a09da2 on tablet 0bfcc65d57eb4c3597a761f878a09da2 with peer b4105a6d531d401faadf67b894fec79d
I20240501 14:09:17.046865 10481 tablet_copy_client.cc:523] T ae06bed9bd844a05ae2ac69148bf5bad P 9faf312806f54f2e8a4a807b234e0d1b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:17.048223 10481 tablet_bootstrap.cc:492] T ae06bed9bd844a05ae2ac69148bf5bad P 9faf312806f54f2e8a4a807b234e0d1b: Bootstrap starting.
W20240501 14:09:17.061828 7355 catalog_manager.cc:4573] TS ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet ae06bed9bd844a05ae2ac69148bf5bad: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111)
I20240501 14:09:17.068025 10481 tablet_bootstrap.cc:492] T ae06bed9bd844a05ae2ac69148bf5bad P 9faf312806f54f2e8a4a807b234e0d1b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:17.068302 10481 tablet_bootstrap.cc:492] T ae06bed9bd844a05ae2ac69148bf5bad P 9faf312806f54f2e8a4a807b234e0d1b: Bootstrap complete.
I20240501 14:09:17.068483 10481 ts_tablet_manager.cc:1367] T ae06bed9bd844a05ae2ac69148bf5bad P 9faf312806f54f2e8a4a807b234e0d1b: Time spent bootstrapping tablet: real 0.020s user 0.018s sys 0.002s
I20240501 14:09:17.069015 10481 raft_consensus.cc:348] T ae06bed9bd844a05ae2ac69148bf5bad P 9faf312806f54f2e8a4a807b234e0d1b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: true } }
I20240501 14:09:17.069129 10481 raft_consensus.cc:729] T ae06bed9bd844a05ae2ac69148bf5bad P 9faf312806f54f2e8a4a807b234e0d1b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9faf312806f54f2e8a4a807b234e0d1b, State: Initialized, Role: LEARNER
I20240501 14:09:17.069228 10481 consensus_queue.cc:260] T ae06bed9bd844a05ae2ac69148bf5bad P 9faf312806f54f2e8a4a807b234e0d1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: true } }
I20240501 14:09:17.069537 10481 ts_tablet_manager.cc:1398] T ae06bed9bd844a05ae2ac69148bf5bad P 9faf312806f54f2e8a4a807b234e0d1b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:09:17.070111 7662 tablet_copy_service.cc:342] P 9e44efa72e7d4421b284ae74d1cb9634: Request end of tablet copy session 9faf312806f54f2e8a4a807b234e0d1b-ae06bed9bd844a05ae2ac69148bf5bad received from {username='slave'} at 127.7.39.133:34271
I20240501 14:09:17.070205 7662 tablet_copy_service.cc:433] P 9e44efa72e7d4421b284ae74d1cb9634: ending tablet copy session 9faf312806f54f2e8a4a807b234e0d1b-ae06bed9bd844a05ae2ac69148bf5bad on tablet ae06bed9bd844a05ae2ac69148bf5bad with peer 9faf312806f54f2e8a4a807b234e0d1b
I20240501 14:09:17.097148 9215 raft_consensus.cc:1203] T 0bfcc65d57eb4c3597a761f878a09da2 P b4105a6d531d401faadf67b894fec79d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:17.126683 10489 tablet_copy_client.cc:523] T c747bd6504c540cebded0019d82495b4 P 9d7c8c0f7907428eab22633702ead7ab: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:17.128044 10489 tablet_bootstrap.cc:492] T c747bd6504c540cebded0019d82495b4 P 9d7c8c0f7907428eab22633702ead7ab: Bootstrap starting.
I20240501 14:09:17.148072 10489 tablet_bootstrap.cc:492] T c747bd6504c540cebded0019d82495b4 P 9d7c8c0f7907428eab22633702ead7ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:17.148411 10489 tablet_bootstrap.cc:492] T c747bd6504c540cebded0019d82495b4 P 9d7c8c0f7907428eab22633702ead7ab: Bootstrap complete.
I20240501 14:09:17.148592 10489 ts_tablet_manager.cc:1367] T c747bd6504c540cebded0019d82495b4 P 9d7c8c0f7907428eab22633702ead7ab: Time spent bootstrapping tablet: real 0.021s user 0.020s sys 0.000s
I20240501 14:09:17.149165 10489 raft_consensus.cc:348] T c747bd6504c540cebded0019d82495b4 P 9d7c8c0f7907428eab22633702ead7ab [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: NON_VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: true } }
I20240501 14:09:17.149313 10489 raft_consensus.cc:729] T c747bd6504c540cebded0019d82495b4 P 9d7c8c0f7907428eab22633702ead7ab [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9d7c8c0f7907428eab22633702ead7ab, State: Initialized, Role: LEARNER
I20240501 14:09:17.149471 10489 consensus_queue.cc:260] T c747bd6504c540cebded0019d82495b4 P 9d7c8c0f7907428eab22633702ead7ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: NON_VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: true } }
I20240501 14:09:17.149829 10489 ts_tablet_manager.cc:1398] T c747bd6504c540cebded0019d82495b4 P 9d7c8c0f7907428eab22633702ead7ab: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240501 14:09:17.150393 7923 tablet_copy_service.cc:342] P 47eabc135e9f4237b036097085e729d8: Request end of tablet copy session 9d7c8c0f7907428eab22633702ead7ab-c747bd6504c540cebded0019d82495b4 received from {username='slave'} at 127.7.39.134:48187
I20240501 14:09:17.150488 7923 tablet_copy_service.cc:433] P 47eabc135e9f4237b036097085e729d8: ending tablet copy session 9d7c8c0f7907428eab22633702ead7ab-c747bd6504c540cebded0019d82495b4 on tablet c747bd6504c540cebded0019d82495b4 with peer 9d7c8c0f7907428eab22633702ead7ab
W20240501 14:09:17.204704 7839 consensus_peers.cc:485] T 14e325d8d3184c9192b8cebafb414190 P 47eabc135e9f4237b036097085e729d8 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:09:17.254240 7839 consensus_peers.cc:485] T 4d282d79e1664f7ea8912e0c5166eec3 P 47eabc135e9f4237b036097085e729d8 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:09:17.449287 8953 raft_consensus.cc:1203] T ae06bed9bd844a05ae2ac69148bf5bad P 9faf312806f54f2e8a4a807b234e0d1b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240501 14:09:17.518833 9083 raft_consensus.cc:1203] T c747bd6504c540cebded0019d82495b4 P 9d7c8c0f7907428eab22633702ead7ab [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
W20240501 14:09:17.595763 7708 consensus_peers.cc:485] T dcf673c75a8b48da95806e1c43309dda P 9ae5a87ca07b40a788a42734b24f66c6 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:09:17.628099 10645 raft_consensus.cc:1050] T 0bfcc65d57eb4c3597a761f878a09da2 P 47eabc135e9f4237b036097085e729d8: attempting to promote NON_VOTER b4105a6d531d401faadf67b894fec79d to VOTER
I20240501 14:09:17.628610 10645 consensus_queue.cc:237] T 0bfcc65d57eb4c3597a761f878a09da2 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } }
I20240501 14:09:17.630080 9215 raft_consensus.cc:1261] T 0bfcc65d57eb4c3597a761f878a09da2 P b4105a6d531d401faadf67b894fec79d [term 1 LEARNER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:17.630326 7641 raft_consensus.cc:1261] T 0bfcc65d57eb4c3597a761f878a09da2 P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 FOLLOWER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:17.630450 10645 consensus_queue.cc:1035] T 0bfcc65d57eb4c3597a761f878a09da2 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:17.630945 7771 raft_consensus.cc:1261] T 0bfcc65d57eb4c3597a761f878a09da2 P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:17.630954 10647 consensus_queue.cc:1035] T 0bfcc65d57eb4c3597a761f878a09da2 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:17.631323 10645 consensus_queue.cc:1035] T 0bfcc65d57eb4c3597a761f878a09da2 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:17.633320 10647 raft_consensus.cc:2932] T 0bfcc65d57eb4c3597a761f878a09da2 P 47eabc135e9f4237b036097085e729d8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b4105a6d531d401faadf67b894fec79d (127.7.39.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } }
I20240501 14:09:17.633777 7771 raft_consensus.cc:2932] T 0bfcc65d57eb4c3597a761f878a09da2 P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b4105a6d531d401faadf67b894fec79d (127.7.39.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } }
I20240501 14:09:17.633973 7641 raft_consensus.cc:2932] T 0bfcc65d57eb4c3597a761f878a09da2 P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b4105a6d531d401faadf67b894fec79d (127.7.39.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } }
I20240501 14:09:17.636937 7364 catalog_manager.cc:5484] T 0bfcc65d57eb4c3597a761f878a09da2 P 9e44efa72e7d4421b284ae74d1cb9634 reported cstate change: config changed from index 4 to 5, b4105a6d531d401faadf67b894fec79d (127.7.39.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "47eabc135e9f4237b036097085e729d8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } }
I20240501 14:09:17.637506 9215 raft_consensus.cc:2932] T 0bfcc65d57eb4c3597a761f878a09da2 P b4105a6d531d401faadf67b894fec79d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b4105a6d531d401faadf67b894fec79d (127.7.39.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } }
I20240501 14:09:17.643147 7904 consensus_queue.cc:237] T 0bfcc65d57eb4c3597a761f878a09da2 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } }
I20240501 14:09:17.644935 9215 raft_consensus.cc:1261] T 0bfcc65d57eb4c3597a761f878a09da2 P b4105a6d531d401faadf67b894fec79d [term 1 FOLLOWER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:17.645277 7771 raft_consensus.cc:1261] T 0bfcc65d57eb4c3597a761f878a09da2 P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:17.645956 10525 consensus_queue.cc:1035] T 0bfcc65d57eb4c3597a761f878a09da2 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:17.646122 10645 consensus_queue.cc:1035] T 0bfcc65d57eb4c3597a761f878a09da2 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:17.648310 10647 raft_consensus.cc:2932] T 0bfcc65d57eb4c3597a761f878a09da2 P 47eabc135e9f4237b036097085e729d8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9e44efa72e7d4421b284ae74d1cb9634 (127.7.39.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } }
I20240501 14:09:17.649776 7354 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0bfcc65d57eb4c3597a761f878a09da2 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:17.650307 9215 raft_consensus.cc:2932] T 0bfcc65d57eb4c3597a761f878a09da2 P b4105a6d531d401faadf67b894fec79d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9e44efa72e7d4421b284ae74d1cb9634 (127.7.39.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } }
I20240501 14:09:17.650861 7365 catalog_manager.cc:5484] T 0bfcc65d57eb4c3597a761f878a09da2 P 47eabc135e9f4237b036097085e729d8 reported cstate change: config changed from index 5 to 6, VOTER 9e44efa72e7d4421b284ae74d1cb9634 (127.7.39.130) evicted. New cstate: current_term: 1 leader_uuid: "47eabc135e9f4237b036097085e729d8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:17.652701 7771 raft_consensus.cc:2932] T 0bfcc65d57eb4c3597a761f878a09da2 P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9e44efa72e7d4421b284ae74d1cb9634 (127.7.39.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } }
I20240501 14:09:17.654289 7620 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bfcc65d57eb4c3597a761f878a09da2 with delete_type TABLET_DATA_TOMBSTONED (TS 9e44efa72e7d4421b284ae74d1cb9634 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60602
I20240501 14:09:17.655565 10667 tablet_replica.cc:330] T 0bfcc65d57eb4c3597a761f878a09da2 P 9e44efa72e7d4421b284ae74d1cb9634: stopping tablet replica
I20240501 14:09:17.660835 10667 raft_consensus.cc:2229] T 0bfcc65d57eb4c3597a761f878a09da2 P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:17.661302 10667 raft_consensus.cc:2258] T 0bfcc65d57eb4c3597a761f878a09da2 P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:17.663130 10667 ts_tablet_manager.cc:1875] T 0bfcc65d57eb4c3597a761f878a09da2 P 9e44efa72e7d4421b284ae74d1cb9634: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:17.667984 10667 ts_tablet_manager.cc:1888] T 0bfcc65d57eb4c3597a761f878a09da2 P 9e44efa72e7d4421b284ae74d1cb9634: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:09:17.668140 10667 log.cc:1196] T 0bfcc65d57eb4c3597a761f878a09da2 P 9e44efa72e7d4421b284ae74d1cb9634: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal/wals/0bfcc65d57eb4c3597a761f878a09da2
I20240501 14:09:17.668792 7355 catalog_manager.cc:4830] TS 9e44efa72e7d4421b284ae74d1cb9634 (127.7.39.130:46433): tablet 0bfcc65d57eb4c3597a761f878a09da2 (table rebalance_test_table_1 [id=95829980e415451998b6e8797247120f]) successfully deleted
W20240501 14:09:17.879076 7708 consensus_peers.cc:485] T ceee52d679bf4d8e8896a6c0dde0519b P 9ae5a87ca07b40a788a42734b24f66c6 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:09:18.026517 10700 raft_consensus.cc:1050] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634: attempting to promote NON_VOTER 9faf312806f54f2e8a4a807b234e0d1b to VOTER
I20240501 14:09:18.027371 10700 consensus_queue.cc:237] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } }
I20240501 14:09:18.028724 9083 raft_consensus.cc:1261] T ae06bed9bd844a05ae2ac69148bf5bad P 9d7c8c0f7907428eab22633702ead7ab [term 1 FOLLOWER]: Refusing update from remote peer 9e44efa72e7d4421b284ae74d1cb9634: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:18.028820 8953 raft_consensus.cc:1261] T ae06bed9bd844a05ae2ac69148bf5bad P 9faf312806f54f2e8a4a807b234e0d1b [term 1 LEARNER]: Refusing update from remote peer 9e44efa72e7d4421b284ae74d1cb9634: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:18.028846 7771 raft_consensus.cc:1261] T ae06bed9bd844a05ae2ac69148bf5bad P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Refusing update from remote peer 9e44efa72e7d4421b284ae74d1cb9634: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:18.030084 10609 consensus_queue.cc:1035] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:18.030267 10703 consensus_queue.cc:1035] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:18.030493 10700 consensus_queue.cc:1035] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:18.032465 10609 raft_consensus.cc:2932] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9faf312806f54f2e8a4a807b234e0d1b (127.7.39.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } }
I20240501 14:09:18.032876 8953 raft_consensus.cc:2932] T ae06bed9bd844a05ae2ac69148bf5bad P 9faf312806f54f2e8a4a807b234e0d1b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9faf312806f54f2e8a4a807b234e0d1b (127.7.39.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } }
I20240501 14:09:18.032876 9083 raft_consensus.cc:2932] T ae06bed9bd844a05ae2ac69148bf5bad P 9d7c8c0f7907428eab22633702ead7ab [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9faf312806f54f2e8a4a807b234e0d1b (127.7.39.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } }
I20240501 14:09:18.036227 7771 raft_consensus.cc:2932] T ae06bed9bd844a05ae2ac69148bf5bad P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9faf312806f54f2e8a4a807b234e0d1b (127.7.39.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } }
I20240501 14:09:18.036998 7367 catalog_manager.cc:5484] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 reported cstate change: config changed from index 8 to 9, 9faf312806f54f2e8a4a807b234e0d1b (127.7.39.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9e44efa72e7d4421b284ae74d1cb9634" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:18.041649 7641 consensus_queue.cc:237] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } }
I20240501 14:09:18.042891 8953 raft_consensus.cc:1261] T ae06bed9bd844a05ae2ac69148bf5bad P 9faf312806f54f2e8a4a807b234e0d1b [term 1 FOLLOWER]: Refusing update from remote peer 9e44efa72e7d4421b284ae74d1cb9634: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:09:18.043280 9083 raft_consensus.cc:1261] T ae06bed9bd844a05ae2ac69148bf5bad P 9d7c8c0f7907428eab22633702ead7ab [term 1 FOLLOWER]: Refusing update from remote peer 9e44efa72e7d4421b284ae74d1cb9634: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:09:18.043355 10703 consensus_queue.cc:1035] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:09:18.043666 10700 consensus_queue.cc:1035] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:09:18.044909 10703 raft_consensus.cc:2932] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 9ae5a87ca07b40a788a42734b24f66c6 (127.7.39.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } }
I20240501 14:09:18.045339 8953 raft_consensus.cc:2932] T ae06bed9bd844a05ae2ac69148bf5bad P 9faf312806f54f2e8a4a807b234e0d1b [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 9ae5a87ca07b40a788a42734b24f66c6 (127.7.39.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } }
I20240501 14:09:18.045645 9083 raft_consensus.cc:2932] T ae06bed9bd844a05ae2ac69148bf5bad P 9d7c8c0f7907428eab22633702ead7ab [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 9ae5a87ca07b40a788a42734b24f66c6 (127.7.39.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } }
I20240501 14:09:18.046527 7355 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ae06bed9bd844a05ae2ac69148bf5bad with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:18.047739 7364 catalog_manager.cc:5484] T ae06bed9bd844a05ae2ac69148bf5bad P 9e44efa72e7d4421b284ae74d1cb9634 reported cstate change: config changed from index 9 to 10, VOTER 9ae5a87ca07b40a788a42734b24f66c6 (127.7.39.131) evicted. New cstate: current_term: 1 leader_uuid: "9e44efa72e7d4421b284ae74d1cb9634" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:18.051136 7752 tablet_service.cc:1514] Processing DeleteTablet for tablet ae06bed9bd844a05ae2ac69148bf5bad with delete_type TABLET_DATA_TOMBSTONED (TS 9ae5a87ca07b40a788a42734b24f66c6 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:36366
I20240501 14:09:18.051656 10709 tablet_replica.cc:330] T ae06bed9bd844a05ae2ac69148bf5bad P 9ae5a87ca07b40a788a42734b24f66c6: stopping tablet replica
I20240501 14:09:18.051959 10709 raft_consensus.cc:2229] T ae06bed9bd844a05ae2ac69148bf5bad P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:18.052237 10709 raft_consensus.cc:2258] T ae06bed9bd844a05ae2ac69148bf5bad P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:18.053499 10709 ts_tablet_manager.cc:1875] T ae06bed9bd844a05ae2ac69148bf5bad P 9ae5a87ca07b40a788a42734b24f66c6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:18.055713 10709 ts_tablet_manager.cc:1888] T ae06bed9bd844a05ae2ac69148bf5bad P 9ae5a87ca07b40a788a42734b24f66c6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 14:09:18.055822 10709 log.cc:1196] T ae06bed9bd844a05ae2ac69148bf5bad P 9ae5a87ca07b40a788a42734b24f66c6: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal/wals/ae06bed9bd844a05ae2ac69148bf5bad
I20240501 14:09:18.056432 7355 catalog_manager.cc:4830] TS 9ae5a87ca07b40a788a42734b24f66c6 (127.7.39.131:34697): tablet ae06bed9bd844a05ae2ac69148bf5bad (table rebalance_test_table_1 [id=95829980e415451998b6e8797247120f]) successfully deleted
I20240501 14:09:18.068591 10525 raft_consensus.cc:1050] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8: attempting to promote NON_VOTER 9d7c8c0f7907428eab22633702ead7ab to VOTER
I20240501 14:09:18.069087 10525 consensus_queue.cc:237] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } }
I20240501 14:09:18.070530 9083 raft_consensus.cc:1261] T c747bd6504c540cebded0019d82495b4 P 9d7c8c0f7907428eab22633702ead7ab [term 1 LEARNER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:18.070581 7771 raft_consensus.cc:1261] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:18.070698 9346 raft_consensus.cc:1261] T c747bd6504c540cebded0019d82495b4 P 9d32e843754e44d0b9aae6f472a4d9a2 [term 1 FOLLOWER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:18.070870 10525 consensus_queue.cc:1035] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:18.071116 10647 consensus_queue.cc:1035] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:18.071349 10645 consensus_queue.cc:1035] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:18.073078 10647 raft_consensus.cc:2932] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9d7c8c0f7907428eab22633702ead7ab (127.7.39.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } }
I20240501 14:09:18.073814 7771 raft_consensus.cc:2932] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9d7c8c0f7907428eab22633702ead7ab (127.7.39.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } }
I20240501 14:09:18.074671 9083 raft_consensus.cc:2932] T c747bd6504c540cebded0019d82495b4 P 9d7c8c0f7907428eab22633702ead7ab [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9d7c8c0f7907428eab22633702ead7ab (127.7.39.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } }
I20240501 14:09:18.075604 7366 catalog_manager.cc:5484] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 reported cstate change: config changed from index 8 to 9, 9d7c8c0f7907428eab22633702ead7ab (127.7.39.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "47eabc135e9f4237b036097085e729d8" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:18.076579 9346 raft_consensus.cc:2932] T c747bd6504c540cebded0019d82495b4 P 9d32e843754e44d0b9aae6f472a4d9a2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9d7c8c0f7907428eab22633702ead7ab (127.7.39.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } }
I20240501 14:09:18.116959 7903 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c747bd6504c540cebded0019d82495b4"
dest_uuid: "47eabc135e9f4237b036097085e729d8"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51842
I20240501 14:09:18.117136 7903 raft_consensus.cc:595] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:18.125175 7641 consensus_queue.cc:237] T 4573be69270c489289af5fc65f67112d P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: NON_VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: true } }
I20240501 14:09:18.126921 7771 raft_consensus.cc:1261] T 4573be69270c489289af5fc65f67112d P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Refusing update from remote peer 9e44efa72e7d4421b284ae74d1cb9634: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:18.127089 7903 raft_consensus.cc:1261] T 4573be69270c489289af5fc65f67112d P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Refusing update from remote peer 9e44efa72e7d4421b284ae74d1cb9634: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:18.127337 10701 consensus_queue.cc:1035] T 4573be69270c489289af5fc65f67112d P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:18.127530 10609 consensus_queue.cc:1035] T 4573be69270c489289af5fc65f67112d P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:18.129464 10703 raft_consensus.cc:2932] T 4573be69270c489289af5fc65f67112d P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9d32e843754e44d0b9aae6f472a4d9a2 (127.7.39.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: NON_VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: true } } }
I20240501 14:09:18.129865 7903 raft_consensus.cc:2932] T 4573be69270c489289af5fc65f67112d P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9d32e843754e44d0b9aae6f472a4d9a2 (127.7.39.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: NON_VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: true } } }
W20240501 14:09:18.130887 7576 consensus_peers.cc:485] T 4573be69270c489289af5fc65f67112d P 9e44efa72e7d4421b284ae74d1cb9634 -> Peer 9d32e843754e44d0b9aae6f472a4d9a2 (127.7.39.136:37475): Couldn't send request to peer 9d32e843754e44d0b9aae6f472a4d9a2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4573be69270c489289af5fc65f67112d. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:18.131855 7771 raft_consensus.cc:2932] T 4573be69270c489289af5fc65f67112d P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9d32e843754e44d0b9aae6f472a4d9a2 (127.7.39.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: NON_VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: true } } }
I20240501 14:09:18.132586 7365 catalog_manager.cc:5484] T 4573be69270c489289af5fc65f67112d P 47eabc135e9f4237b036097085e729d8 reported cstate change: config changed from index -1 to 4, NON_VOTER 9d32e843754e44d0b9aae6f472a4d9a2 (127.7.39.136) added. New cstate: current_term: 1 leader_uuid: "9e44efa72e7d4421b284ae74d1cb9634" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: NON_VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: true } } }
I20240501 14:09:18.149279 7903 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c747bd6504c540cebded0019d82495b4"
dest_uuid: "47eabc135e9f4237b036097085e729d8"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51848
I20240501 14:09:18.149418 7903 raft_consensus.cc:595] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [term 1 LEADER]: Received request to transfer leadership
W20240501 14:09:18.277832 7708 consensus_peers.cc:485] T 2880045de45743bdb5358a17d93b2245 P 9ae5a87ca07b40a788a42734b24f66c6 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:09:18.366065 7903 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c747bd6504c540cebded0019d82495b4"
dest_uuid: "47eabc135e9f4237b036097085e729d8"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51878
I20240501 14:09:18.366222 7903 raft_consensus.cc:595] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:18.490053 10645 raft_consensus.cc:979] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8: : Instructing follower 9ae5a87ca07b40a788a42734b24f66c6 to start an election
I20240501 14:09:18.490170 10645 raft_consensus.cc:1067] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [term 1 LEADER]: Signalling peer 9ae5a87ca07b40a788a42734b24f66c6 to start an election
I20240501 14:09:18.490742 7771 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c747bd6504c540cebded0019d82495b4"
dest_uuid: "9ae5a87ca07b40a788a42734b24f66c6"
from {username='slave'} at 127.7.39.132:39797
I20240501 14:09:18.490902 7771 raft_consensus.cc:482] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:18.490965 7771 raft_consensus.cc:3037] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:18.491801 7771 raft_consensus.cc:504] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } }
I20240501 14:09:18.492399 7771 leader_election.cc:288] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [CANDIDATE]: Term 2 election: Requested vote from peers 47eabc135e9f4237b036097085e729d8 (127.7.39.132:42369), 9d32e843754e44d0b9aae6f472a4d9a2 (127.7.39.136:37475), 9d7c8c0f7907428eab22633702ead7ab (127.7.39.134:38699)
I20240501 14:09:18.492893 7903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c747bd6504c540cebded0019d82495b4" candidate_uuid: "9ae5a87ca07b40a788a42734b24f66c6" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "47eabc135e9f4237b036097085e729d8"
I20240501 14:09:18.493039 7903 raft_consensus.cc:3032] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:09:18.493084 7903 raft_consensus.cc:729] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 47eabc135e9f4237b036097085e729d8, State: Running, Role: LEADER
I20240501 14:09:18.493249 7903 consensus_queue.cc:260] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } }
I20240501 14:09:18.493444 7903 raft_consensus.cc:3037] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:18.494400 7903 raft_consensus.cc:2454] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ae5a87ca07b40a788a42734b24f66c6 in term 2.
I20240501 14:09:18.496502 9346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c747bd6504c540cebded0019d82495b4" candidate_uuid: "9ae5a87ca07b40a788a42734b24f66c6" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "9d32e843754e44d0b9aae6f472a4d9a2"
I20240501 14:09:18.496693 9346 raft_consensus.cc:3037] T c747bd6504c540cebded0019d82495b4 P 9d32e843754e44d0b9aae6f472a4d9a2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:18.497176 9083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c747bd6504c540cebded0019d82495b4" candidate_uuid: "9ae5a87ca07b40a788a42734b24f66c6" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "9d7c8c0f7907428eab22633702ead7ab"
I20240501 14:09:18.497357 9083 raft_consensus.cc:3037] T c747bd6504c540cebded0019d82495b4 P 9d7c8c0f7907428eab22633702ead7ab [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:18.497843 9346 raft_consensus.cc:2454] T c747bd6504c540cebded0019d82495b4 P 9d32e843754e44d0b9aae6f472a4d9a2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ae5a87ca07b40a788a42734b24f66c6 in term 2.
I20240501 14:09:18.498160 7707 leader_election.cc:302] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 47eabc135e9f4237b036097085e729d8, 9ae5a87ca07b40a788a42734b24f66c6, 9d32e843754e44d0b9aae6f472a4d9a2; no voters:
I20240501 14:09:18.498337 10036 raft_consensus.cc:2781] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:18.498371 9083 raft_consensus.cc:2454] T c747bd6504c540cebded0019d82495b4 P 9d7c8c0f7907428eab22633702ead7ab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ae5a87ca07b40a788a42734b24f66c6 in term 2.
I20240501 14:09:18.498533 10036 raft_consensus.cc:686] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [term 2 LEADER]: Becoming Leader. State: Replica: 9ae5a87ca07b40a788a42734b24f66c6, State: Running, Role: LEADER
I20240501 14:09:18.498728 10036 consensus_queue.cc:237] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } }
I20240501 14:09:18.500489 7364 catalog_manager.cc:5484] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 reported cstate change: term changed from 1 to 2, leader changed from 47eabc135e9f4237b036097085e729d8 (127.7.39.132) to 9ae5a87ca07b40a788a42734b24f66c6 (127.7.39.131). New cstate: current_term: 2 leader_uuid: "9ae5a87ca07b40a788a42734b24f66c6" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:18.742449 7355 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20240501 14:09:18.743232 7355 catalog_manager.cc:4573] TS ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 2112bf96dc3343678aa6f46bf2a3b1f2: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111)
I20240501 14:09:18.751091 10785 ts_tablet_manager.cc:902] T 4573be69270c489289af5fc65f67112d P 9d32e843754e44d0b9aae6f472a4d9a2: Initiating tablet copy from peer 9e44efa72e7d4421b284ae74d1cb9634 (127.7.39.130:46433)
I20240501 14:09:18.751611 10785 tablet_copy_client.cc:308] T 4573be69270c489289af5fc65f67112d P 9d32e843754e44d0b9aae6f472a4d9a2: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.130:46433
I20240501 14:09:18.754812 7662 tablet_copy_service.cc:140] P 9e44efa72e7d4421b284ae74d1cb9634: Received BeginTabletCopySession request for tablet 4573be69270c489289af5fc65f67112d from peer 9d32e843754e44d0b9aae6f472a4d9a2 ({username='slave'} at 127.7.39.136:48105)
I20240501 14:09:18.754906 7662 tablet_copy_service.cc:161] P 9e44efa72e7d4421b284ae74d1cb9634: Beginning new tablet copy session on tablet 4573be69270c489289af5fc65f67112d from peer 9d32e843754e44d0b9aae6f472a4d9a2 at {username='slave'} at 127.7.39.136:48105: session id = 9d32e843754e44d0b9aae6f472a4d9a2-4573be69270c489289af5fc65f67112d
I20240501 14:09:18.755681 7662 tablet_copy_source_session.cc:205] T 4573be69270c489289af5fc65f67112d P 9e44efa72e7d4421b284ae74d1cb9634: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:18.756525 10785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4573be69270c489289af5fc65f67112d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:18.758713 10785 tablet_copy_client.cc:783] T 4573be69270c489289af5fc65f67112d P 9d32e843754e44d0b9aae6f472a4d9a2: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:18.758875 10785 tablet_copy_client.cc:647] T 4573be69270c489289af5fc65f67112d P 9d32e843754e44d0b9aae6f472a4d9a2: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:18.759865 10789 tablet_copy_client.cc:1002] T 4573be69270c489289af5fc65f67112d P 9d32e843754e44d0b9aae6f472a4d9a2: tablet copy: Injecting latency into file download: 1500
I20240501 14:09:19.002548 9083 raft_consensus.cc:1261] T c747bd6504c540cebded0019d82495b4 P 9d7c8c0f7907428eab22633702ead7ab [term 2 FOLLOWER]: Refusing update from remote peer 9ae5a87ca07b40a788a42734b24f66c6: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 14:09:19.002979 10771 consensus_queue.cc:1035] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240501 14:09:19.009193 7577 consensus_peers.cc:485] T 2cf2f85819584fdf872f6c590d11f873 P 9e44efa72e7d4421b284ae74d1cb9634 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:09:19.066646 7903 raft_consensus.cc:1261] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [term 2 FOLLOWER]: Refusing update from remote peer 9ae5a87ca07b40a788a42734b24f66c6: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 14:09:19.067101 10771 consensus_queue.cc:1035] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:09:19.068534 9346 raft_consensus.cc:1261] T c747bd6504c540cebded0019d82495b4 P 9d32e843754e44d0b9aae6f472a4d9a2 [term 2 FOLLOWER]: Refusing update from remote peer 9ae5a87ca07b40a788a42734b24f66c6: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 14:09:19.069581 10813 consensus_queue.cc:1035] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240501 14:09:19.401487 7577 consensus_peers.cc:485] T df856440b07b418394477cf368b765c1 P 9e44efa72e7d4421b284ae74d1cb9634 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:09:19.564749 7771 consensus_queue.cc:237] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } }
I20240501 14:09:19.565878 9083 raft_consensus.cc:1261] T c747bd6504c540cebded0019d82495b4 P 9d7c8c0f7907428eab22633702ead7ab [term 2 FOLLOWER]: Refusing update from remote peer 9ae5a87ca07b40a788a42734b24f66c6: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 14:09:19.565892 9346 raft_consensus.cc:1261] T c747bd6504c540cebded0019d82495b4 P 9d32e843754e44d0b9aae6f472a4d9a2 [term 2 FOLLOWER]: Refusing update from remote peer 9ae5a87ca07b40a788a42734b24f66c6: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 14:09:19.566310 10813 consensus_queue.cc:1035] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:09:19.566512 10036 consensus_queue.cc:1035] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:09:19.568120 10771 raft_consensus.cc:2932] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 47eabc135e9f4237b036097085e729d8 (127.7.39.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } }
I20240501 14:09:19.568660 9083 raft_consensus.cc:2932] T c747bd6504c540cebded0019d82495b4 P 9d7c8c0f7907428eab22633702ead7ab [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 47eabc135e9f4237b036097085e729d8 (127.7.39.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } }
I20240501 14:09:19.568678 9346 raft_consensus.cc:2932] T c747bd6504c540cebded0019d82495b4 P 9d32e843754e44d0b9aae6f472a4d9a2 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 47eabc135e9f4237b036097085e729d8 (127.7.39.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } }
I20240501 14:09:19.570303 7355 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c747bd6504c540cebded0019d82495b4 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:19.570972 7364 catalog_manager.cc:5484] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 reported cstate change: config changed from index 9 to 11, VOTER 47eabc135e9f4237b036097085e729d8 (127.7.39.132) evicted. New cstate: current_term: 2 leader_uuid: "9ae5a87ca07b40a788a42734b24f66c6" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:19.573874 7882 tablet_service.cc:1514] Processing DeleteTablet for tablet c747bd6504c540cebded0019d82495b4 with delete_type TABLET_DATA_TOMBSTONED (TS 47eabc135e9f4237b036097085e729d8 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:43764
I20240501 14:09:19.574348 10845 tablet_replica.cc:330] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8: stopping tablet replica
I20240501 14:09:19.574666 10845 raft_consensus.cc:2229] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:19.574882 10845 raft_consensus.cc:2258] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:19.575922 10845 ts_tablet_manager.cc:1875] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:19.577857 10845 ts_tablet_manager.cc:1888] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240501 14:09:19.577930 10845 log.cc:1196] T c747bd6504c540cebded0019d82495b4 P 47eabc135e9f4237b036097085e729d8: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal/wals/c747bd6504c540cebded0019d82495b4
I20240501 14:09:19.578444 7354 catalog_manager.cc:4830] TS 47eabc135e9f4237b036097085e729d8 (127.7.39.132:42369): tablet c747bd6504c540cebded0019d82495b4 (table rebalance_test_table_4 [id=9b4ecb4938fd4bc185e20ce3ec5254f5]) successfully deleted
I20240501 14:09:19.646806 7903 consensus_queue.cc:237] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: NON_VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: true } }
I20240501 14:09:19.648402 9215 raft_consensus.cc:1261] T 3137e7904ad04fc79d4028c1882e2412 P b4105a6d531d401faadf67b894fec79d [term 2 FOLLOWER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
W20240501 14:09:19.648507 7840 consensus_peers.cc:485] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 -> Peer 9d7c8c0f7907428eab22633702ead7ab (127.7.39.134:38699): Couldn't send request to peer 9d7c8c0f7907428eab22633702ead7ab. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3137e7904ad04fc79d4028c1882e2412. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:19.648535 7641 raft_consensus.cc:1261] T 3137e7904ad04fc79d4028c1882e2412 P 9e44efa72e7d4421b284ae74d1cb9634 [term 2 FOLLOWER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:09:19.648808 10864 consensus_queue.cc:1035] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:19.648975 10645 consensus_queue.cc:1035] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:19.650697 10863 raft_consensus.cc:2932] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 9d7c8c0f7907428eab22633702ead7ab (127.7.39.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: NON_VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: true } } }
I20240501 14:09:19.651155 7641 raft_consensus.cc:2932] T 3137e7904ad04fc79d4028c1882e2412 P 9e44efa72e7d4421b284ae74d1cb9634 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 9d7c8c0f7907428eab22633702ead7ab (127.7.39.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: NON_VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: true } } }
I20240501 14:09:19.651216 9215 raft_consensus.cc:2932] T 3137e7904ad04fc79d4028c1882e2412 P b4105a6d531d401faadf67b894fec79d [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 9d7c8c0f7907428eab22633702ead7ab (127.7.39.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: NON_VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: true } } }
I20240501 14:09:19.653690 7366 catalog_manager.cc:5484] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 reported cstate change: config changed from index 8 to 9, NON_VOTER 9d7c8c0f7907428eab22633702ead7ab (127.7.39.134) added. New cstate: current_term: 2 leader_uuid: "47eabc135e9f4237b036097085e729d8" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: NON_VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:19.661726 7903 consensus_queue.cc:237] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: true } }
I20240501 14:09:19.663480 9215 raft_consensus.cc:1261] T 2112bf96dc3343678aa6f46bf2a3b1f2 P b4105a6d531d401faadf67b894fec79d [term 1 FOLLOWER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:19.663715 7771 raft_consensus.cc:1261] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:19.663870 10645 consensus_queue.cc:1035] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:19.664176 10863 consensus_queue.cc:1035] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:19.665654 10645 raft_consensus.cc:2932] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9faf312806f54f2e8a4a807b234e0d1b (127.7.39.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: true } } }
I20240501 14:09:19.666018 9215 raft_consensus.cc:2932] T 2112bf96dc3343678aa6f46bf2a3b1f2 P b4105a6d531d401faadf67b894fec79d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9faf312806f54f2e8a4a807b234e0d1b (127.7.39.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: true } } }
I20240501 14:09:19.667104 7771 raft_consensus.cc:2932] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9faf312806f54f2e8a4a807b234e0d1b (127.7.39.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: true } } }
W20240501 14:09:19.668080 7838 consensus_peers.cc:485] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 -> Peer 9faf312806f54f2e8a4a807b234e0d1b (127.7.39.133:38805): Couldn't send request to peer 9faf312806f54f2e8a4a807b234e0d1b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2112bf96dc3343678aa6f46bf2a3b1f2. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:19.668725 7364 catalog_manager.cc:5484] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 reported cstate change: config changed from index 7 to 8, NON_VOTER 9faf312806f54f2e8a4a807b234e0d1b (127.7.39.133) added. New cstate: current_term: 1 leader_uuid: "47eabc135e9f4237b036097085e729d8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:19.789250 7839 consensus_peers.cc:485] T 14e325d8d3184c9192b8cebafb414190 P 47eabc135e9f4237b036097085e729d8 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:09:19.866427 7839 consensus_peers.cc:485] T 4d282d79e1664f7ea8912e0c5166eec3 P 47eabc135e9f4237b036097085e729d8 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:09:19.917141 7708 consensus_peers.cc:485] T dcf673c75a8b48da95806e1c43309dda P 9ae5a87ca07b40a788a42734b24f66c6 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:09:19.917837 7355 catalog_manager.cc:4573] TS ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet ab7c1f81f7524d55bd43065890088eaf: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111)
I20240501 14:09:20.032469 10909 ts_tablet_manager.cc:902] T 3137e7904ad04fc79d4028c1882e2412 P 9d7c8c0f7907428eab22633702ead7ab: Initiating tablet copy from peer 47eabc135e9f4237b036097085e729d8 (127.7.39.132:42369)
I20240501 14:09:20.033108 10909 tablet_copy_client.cc:308] T 3137e7904ad04fc79d4028c1882e2412 P 9d7c8c0f7907428eab22633702ead7ab: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.132:42369
I20240501 14:09:20.033639 7923 tablet_copy_service.cc:140] P 47eabc135e9f4237b036097085e729d8: Received BeginTabletCopySession request for tablet 3137e7904ad04fc79d4028c1882e2412 from peer 9d7c8c0f7907428eab22633702ead7ab ({username='slave'} at 127.7.39.134:48187)
I20240501 14:09:20.033746 7923 tablet_copy_service.cc:161] P 47eabc135e9f4237b036097085e729d8: Beginning new tablet copy session on tablet 3137e7904ad04fc79d4028c1882e2412 from peer 9d7c8c0f7907428eab22633702ead7ab at {username='slave'} at 127.7.39.134:48187: session id = 9d7c8c0f7907428eab22633702ead7ab-3137e7904ad04fc79d4028c1882e2412
I20240501 14:09:20.034613 7923 tablet_copy_source_session.cc:205] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:20.035576 10909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3137e7904ad04fc79d4028c1882e2412. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:20.038009 10909 tablet_copy_client.cc:783] T 3137e7904ad04fc79d4028c1882e2412 P 9d7c8c0f7907428eab22633702ead7ab: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:20.038172 10909 tablet_copy_client.cc:647] T 3137e7904ad04fc79d4028c1882e2412 P 9d7c8c0f7907428eab22633702ead7ab: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:20.039335 10913 tablet_copy_client.cc:1002] T 3137e7904ad04fc79d4028c1882e2412 P 9d7c8c0f7907428eab22633702ead7ab: tablet copy: Injecting latency into file download: 1500
I20240501 14:09:20.107903 10914 ts_tablet_manager.cc:902] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9faf312806f54f2e8a4a807b234e0d1b: Initiating tablet copy from peer 47eabc135e9f4237b036097085e729d8 (127.7.39.132:42369)
I20240501 14:09:20.108364 10914 tablet_copy_client.cc:308] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9faf312806f54f2e8a4a807b234e0d1b: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.132:42369
I20240501 14:09:20.112573 7923 tablet_copy_service.cc:140] P 47eabc135e9f4237b036097085e729d8: Received BeginTabletCopySession request for tablet 2112bf96dc3343678aa6f46bf2a3b1f2 from peer 9faf312806f54f2e8a4a807b234e0d1b ({username='slave'} at 127.7.39.133:39467)
I20240501 14:09:20.112690 7923 tablet_copy_service.cc:161] P 47eabc135e9f4237b036097085e729d8: Beginning new tablet copy session on tablet 2112bf96dc3343678aa6f46bf2a3b1f2 from peer 9faf312806f54f2e8a4a807b234e0d1b at {username='slave'} at 127.7.39.133:39467: session id = 9faf312806f54f2e8a4a807b234e0d1b-2112bf96dc3343678aa6f46bf2a3b1f2
I20240501 14:09:20.113587 7923 tablet_copy_source_session.cc:205] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:20.114467 10914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2112bf96dc3343678aa6f46bf2a3b1f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:20.116662 10914 tablet_copy_client.cc:783] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9faf312806f54f2e8a4a807b234e0d1b: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:20.116811 10914 tablet_copy_client.cc:647] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9faf312806f54f2e8a4a807b234e0d1b: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:20.117760 10918 tablet_copy_client.cc:1002] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9faf312806f54f2e8a4a807b234e0d1b: tablet copy: Injecting latency into file download: 1500
W20240501 14:09:20.173501 7577 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 14:09:20.238734 7839 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 14:09:20.260159 10785 tablet_copy_client.cc:523] T 4573be69270c489289af5fc65f67112d P 9d32e843754e44d0b9aae6f472a4d9a2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:20.261513 10785 tablet_bootstrap.cc:492] T 4573be69270c489289af5fc65f67112d P 9d32e843754e44d0b9aae6f472a4d9a2: Bootstrap starting.
I20240501 14:09:20.279839 10785 tablet_bootstrap.cc:492] T 4573be69270c489289af5fc65f67112d P 9d32e843754e44d0b9aae6f472a4d9a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:20.280161 10785 tablet_bootstrap.cc:492] T 4573be69270c489289af5fc65f67112d P 9d32e843754e44d0b9aae6f472a4d9a2: Bootstrap complete.
I20240501 14:09:20.280366 10785 ts_tablet_manager.cc:1367] T 4573be69270c489289af5fc65f67112d P 9d32e843754e44d0b9aae6f472a4d9a2: Time spent bootstrapping tablet: real 0.019s user 0.018s sys 0.000s
I20240501 14:09:20.280872 10785 raft_consensus.cc:348] T 4573be69270c489289af5fc65f67112d P 9d32e843754e44d0b9aae6f472a4d9a2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: NON_VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: true } }
I20240501 14:09:20.280982 10785 raft_consensus.cc:729] T 4573be69270c489289af5fc65f67112d P 9d32e843754e44d0b9aae6f472a4d9a2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9d32e843754e44d0b9aae6f472a4d9a2, State: Initialized, Role: LEARNER
I20240501 14:09:20.281062 10785 consensus_queue.cc:260] T 4573be69270c489289af5fc65f67112d P 9d32e843754e44d0b9aae6f472a4d9a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: NON_VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: true } }
I20240501 14:09:20.281378 10785 ts_tablet_manager.cc:1398] T 4573be69270c489289af5fc65f67112d P 9d32e843754e44d0b9aae6f472a4d9a2: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:09:20.282076 7662 tablet_copy_service.cc:342] P 9e44efa72e7d4421b284ae74d1cb9634: Request end of tablet copy session 9d32e843754e44d0b9aae6f472a4d9a2-4573be69270c489289af5fc65f67112d received from {username='slave'} at 127.7.39.136:48105
I20240501 14:09:20.282178 7662 tablet_copy_service.cc:433] P 9e44efa72e7d4421b284ae74d1cb9634: ending tablet copy session 9d32e843754e44d0b9aae6f472a4d9a2-4573be69270c489289af5fc65f67112d on tablet 4573be69270c489289af5fc65f67112d with peer 9d32e843754e44d0b9aae6f472a4d9a2
W20240501 14:09:20.359603 7708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 14:09:20.381462 7708 consensus_peers.cc:485] T ceee52d679bf4d8e8896a6c0dde0519b P 9ae5a87ca07b40a788a42734b24f66c6 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 14:09:20.678217 9346 raft_consensus.cc:1203] T 4573be69270c489289af5fc65f67112d P 9d32e843754e44d0b9aae6f472a4d9a2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240501 14:09:21.042928 7708 consensus_peers.cc:485] T 2880045de45743bdb5358a17d93b2245 P 9ae5a87ca07b40a788a42734b24f66c6 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 14:09:21.163949 11000 raft_consensus.cc:1050] T 4573be69270c489289af5fc65f67112d P 9e44efa72e7d4421b284ae74d1cb9634: attempting to promote NON_VOTER 9d32e843754e44d0b9aae6f472a4d9a2 to VOTER
I20240501 14:09:21.164345 11000 consensus_queue.cc:237] T 4573be69270c489289af5fc65f67112d P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } }
I20240501 14:09:21.165719 7903 raft_consensus.cc:1261] T 4573be69270c489289af5fc65f67112d P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Refusing update from remote peer 9e44efa72e7d4421b284ae74d1cb9634: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:21.165707 7771 raft_consensus.cc:1261] T 4573be69270c489289af5fc65f67112d P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Refusing update from remote peer 9e44efa72e7d4421b284ae74d1cb9634: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:21.165725 9346 raft_consensus.cc:1261] T 4573be69270c489289af5fc65f67112d P 9d32e843754e44d0b9aae6f472a4d9a2 [term 1 LEARNER]: Refusing update from remote peer 9e44efa72e7d4421b284ae74d1cb9634: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:21.166177 10830 consensus_queue.cc:1035] T 4573be69270c489289af5fc65f67112d P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:21.166366 11002 consensus_queue.cc:1035] T 4573be69270c489289af5fc65f67112d P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:21.166548 11000 consensus_queue.cc:1035] T 4573be69270c489289af5fc65f67112d P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:21.168685 10830 raft_consensus.cc:2932] T 4573be69270c489289af5fc65f67112d P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9d32e843754e44d0b9aae6f472a4d9a2 (127.7.39.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } }
I20240501 14:09:21.169193 7903 raft_consensus.cc:2932] T 4573be69270c489289af5fc65f67112d P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9d32e843754e44d0b9aae6f472a4d9a2 (127.7.39.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } }
I20240501 14:09:21.169181 7771 raft_consensus.cc:2932] T 4573be69270c489289af5fc65f67112d P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9d32e843754e44d0b9aae6f472a4d9a2 (127.7.39.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } }
I20240501 14:09:21.171607 9346 raft_consensus.cc:2932] T 4573be69270c489289af5fc65f67112d P 9d32e843754e44d0b9aae6f472a4d9a2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9d32e843754e44d0b9aae6f472a4d9a2 (127.7.39.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } }
I20240501 14:09:21.173024 7366 catalog_manager.cc:5484] T 4573be69270c489289af5fc65f67112d P 9ae5a87ca07b40a788a42734b24f66c6 reported cstate change: config changed from index 4 to 5, 9d32e843754e44d0b9aae6f472a4d9a2 (127.7.39.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9e44efa72e7d4421b284ae74d1cb9634" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } }
I20240501 14:09:21.177747 7641 consensus_queue.cc:237] T 4573be69270c489289af5fc65f67112d P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } }
I20240501 14:09:21.178880 9346 raft_consensus.cc:1261] T 4573be69270c489289af5fc65f67112d P 9d32e843754e44d0b9aae6f472a4d9a2 [term 1 FOLLOWER]: Refusing update from remote peer 9e44efa72e7d4421b284ae74d1cb9634: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:21.178900 7903 raft_consensus.cc:1261] T 4573be69270c489289af5fc65f67112d P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Refusing update from remote peer 9e44efa72e7d4421b284ae74d1cb9634: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:21.179291 10829 consensus_queue.cc:1035] T 4573be69270c489289af5fc65f67112d P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:21.179436 11000 consensus_queue.cc:1035] T 4573be69270c489289af5fc65f67112d P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:21.181135 11002 raft_consensus.cc:2932] T 4573be69270c489289af5fc65f67112d P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9ae5a87ca07b40a788a42734b24f66c6 (127.7.39.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } }
I20240501 14:09:21.181656 9346 raft_consensus.cc:2932] T 4573be69270c489289af5fc65f67112d P 9d32e843754e44d0b9aae6f472a4d9a2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9ae5a87ca07b40a788a42734b24f66c6 (127.7.39.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } }
I20240501 14:09:21.181699 7903 raft_consensus.cc:2932] T 4573be69270c489289af5fc65f67112d P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9ae5a87ca07b40a788a42734b24f66c6 (127.7.39.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } }
I20240501 14:09:21.183480 7355 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4573be69270c489289af5fc65f67112d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:21.184206 7367 catalog_manager.cc:5484] T 4573be69270c489289af5fc65f67112d P 9d32e843754e44d0b9aae6f472a4d9a2 reported cstate change: config changed from index 5 to 6, VOTER 9ae5a87ca07b40a788a42734b24f66c6 (127.7.39.131) evicted. New cstate: current_term: 1 leader_uuid: "9e44efa72e7d4421b284ae74d1cb9634" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } }
I20240501 14:09:21.186450 7752 tablet_service.cc:1514] Processing DeleteTablet for tablet 4573be69270c489289af5fc65f67112d with delete_type TABLET_DATA_TOMBSTONED (TS 9ae5a87ca07b40a788a42734b24f66c6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36366
I20240501 14:09:21.186832 11010 tablet_replica.cc:330] T 4573be69270c489289af5fc65f67112d P 9ae5a87ca07b40a788a42734b24f66c6: stopping tablet replica
I20240501 14:09:21.187011 11010 raft_consensus.cc:2229] T 4573be69270c489289af5fc65f67112d P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:21.187198 11010 raft_consensus.cc:2258] T 4573be69270c489289af5fc65f67112d P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:21.187774 11010 ts_tablet_manager.cc:1875] T 4573be69270c489289af5fc65f67112d P 9ae5a87ca07b40a788a42734b24f66c6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:21.189560 11010 ts_tablet_manager.cc:1888] T 4573be69270c489289af5fc65f67112d P 9ae5a87ca07b40a788a42734b24f66c6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:09:21.189635 11010 log.cc:1196] T 4573be69270c489289af5fc65f67112d P 9ae5a87ca07b40a788a42734b24f66c6: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal/wals/4573be69270c489289af5fc65f67112d
I20240501 14:09:21.190155 7355 catalog_manager.cc:4830] TS 9ae5a87ca07b40a788a42734b24f66c6 (127.7.39.131:34697): tablet 4573be69270c489289af5fc65f67112d (table rebalance_test_table_1 [id=95829980e415451998b6e8797247120f]) successfully deleted
I20240501 14:09:21.539682 10909 tablet_copy_client.cc:523] T 3137e7904ad04fc79d4028c1882e2412 P 9d7c8c0f7907428eab22633702ead7ab: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:21.541312 10909 tablet_bootstrap.cc:492] T 3137e7904ad04fc79d4028c1882e2412 P 9d7c8c0f7907428eab22633702ead7ab: Bootstrap starting.
I20240501 14:09:21.560091 10909 tablet_bootstrap.cc:492] T 3137e7904ad04fc79d4028c1882e2412 P 9d7c8c0f7907428eab22633702ead7ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:21.560388 10909 tablet_bootstrap.cc:492] T 3137e7904ad04fc79d4028c1882e2412 P 9d7c8c0f7907428eab22633702ead7ab: Bootstrap complete.
I20240501 14:09:21.560564 10909 ts_tablet_manager.cc:1367] T 3137e7904ad04fc79d4028c1882e2412 P 9d7c8c0f7907428eab22633702ead7ab: Time spent bootstrapping tablet: real 0.019s user 0.019s sys 0.000s
I20240501 14:09:21.561005 10909 raft_consensus.cc:348] T 3137e7904ad04fc79d4028c1882e2412 P 9d7c8c0f7907428eab22633702ead7ab [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: NON_VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: true } }
I20240501 14:09:21.561131 10909 raft_consensus.cc:729] T 3137e7904ad04fc79d4028c1882e2412 P 9d7c8c0f7907428eab22633702ead7ab [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 9d7c8c0f7907428eab22633702ead7ab, State: Initialized, Role: LEARNER
I20240501 14:09:21.561266 10909 consensus_queue.cc:260] T 3137e7904ad04fc79d4028c1882e2412 P 9d7c8c0f7907428eab22633702ead7ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: NON_VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: true } }
I20240501 14:09:21.561558 10909 ts_tablet_manager.cc:1398] T 3137e7904ad04fc79d4028c1882e2412 P 9d7c8c0f7907428eab22633702ead7ab: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:09:21.562170 7923 tablet_copy_service.cc:342] P 47eabc135e9f4237b036097085e729d8: Request end of tablet copy session 9d7c8c0f7907428eab22633702ead7ab-3137e7904ad04fc79d4028c1882e2412 received from {username='slave'} at 127.7.39.134:48187
I20240501 14:09:21.562255 7923 tablet_copy_service.cc:433] P 47eabc135e9f4237b036097085e729d8: ending tablet copy session 9d7c8c0f7907428eab22633702ead7ab-3137e7904ad04fc79d4028c1882e2412 on tablet 3137e7904ad04fc79d4028c1882e2412 with peer 9d7c8c0f7907428eab22633702ead7ab
I20240501 14:09:21.618137 10914 tablet_copy_client.cc:523] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9faf312806f54f2e8a4a807b234e0d1b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:21.619767 10914 tablet_bootstrap.cc:492] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9faf312806f54f2e8a4a807b234e0d1b: Bootstrap starting.
I20240501 14:09:21.636847 10914 tablet_bootstrap.cc:492] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9faf312806f54f2e8a4a807b234e0d1b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:21.637135 10914 tablet_bootstrap.cc:492] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9faf312806f54f2e8a4a807b234e0d1b: Bootstrap complete.
I20240501 14:09:21.637312 10914 ts_tablet_manager.cc:1367] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9faf312806f54f2e8a4a807b234e0d1b: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.006s
I20240501 14:09:21.637717 10914 raft_consensus.cc:348] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9faf312806f54f2e8a4a807b234e0d1b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: true } }
I20240501 14:09:21.637825 10914 raft_consensus.cc:729] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9faf312806f54f2e8a4a807b234e0d1b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9faf312806f54f2e8a4a807b234e0d1b, State: Initialized, Role: LEARNER
I20240501 14:09:21.637948 10914 consensus_queue.cc:260] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9faf312806f54f2e8a4a807b234e0d1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: true } }
I20240501 14:09:21.638283 10914 ts_tablet_manager.cc:1398] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9faf312806f54f2e8a4a807b234e0d1b: Time spent starting tablet: real 0.001s user 0.002s sys 0.001s
I20240501 14:09:21.638897 7923 tablet_copy_service.cc:342] P 47eabc135e9f4237b036097085e729d8: Request end of tablet copy session 9faf312806f54f2e8a4a807b234e0d1b-2112bf96dc3343678aa6f46bf2a3b1f2 received from {username='slave'} at 127.7.39.133:39467
I20240501 14:09:21.639009 7923 tablet_copy_service.cc:433] P 47eabc135e9f4237b036097085e729d8: ending tablet copy session 9faf312806f54f2e8a4a807b234e0d1b-2112bf96dc3343678aa6f46bf2a3b1f2 on tablet 2112bf96dc3343678aa6f46bf2a3b1f2 with peer 9faf312806f54f2e8a4a807b234e0d1b
W20240501 14:09:21.738888 7577 consensus_peers.cc:485] T 2cf2f85819584fdf872f6c590d11f873 P 9e44efa72e7d4421b284ae74d1cb9634 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 14:09:21.831635 9083 raft_consensus.cc:1203] T 3137e7904ad04fc79d4028c1882e2412 P 9d7c8c0f7907428eab22633702ead7ab [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20240501 14:09:21.966413 8953 raft_consensus.cc:1203] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9faf312806f54f2e8a4a807b234e0d1b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
W20240501 14:09:22.057186 7577 consensus_peers.cc:485] T df856440b07b418394477cf368b765c1 P 9e44efa72e7d4421b284ae74d1cb9634 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:09:22.124106 7839 consensus_peers.cc:485] T 14e325d8d3184c9192b8cebafb414190 P 47eabc135e9f4237b036097085e729d8 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:09:22.193703 7355 catalog_manager.cc:4573] TS ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 3137e7904ad04fc79d4028c1882e2412: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111)
I20240501 14:09:22.284076 10954 raft_consensus.cc:1050] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8: attempting to promote NON_VOTER 9d7c8c0f7907428eab22633702ead7ab to VOTER
I20240501 14:09:22.284520 10954 consensus_queue.cc:237] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } }
I20240501 14:09:22.286175 7641 raft_consensus.cc:1261] T 3137e7904ad04fc79d4028c1882e2412 P 9e44efa72e7d4421b284ae74d1cb9634 [term 2 FOLLOWER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 14:09:22.286196 9215 raft_consensus.cc:1261] T 3137e7904ad04fc79d4028c1882e2412 P b4105a6d531d401faadf67b894fec79d [term 2 FOLLOWER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 14:09:22.286655 11088 consensus_queue.cc:1035] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:09:22.287082 11070 consensus_queue.cc:1035] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:09:22.288089 9083 raft_consensus.cc:1261] T 3137e7904ad04fc79d4028c1882e2412 P 9d7c8c0f7907428eab22633702ead7ab [term 2 LEARNER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 14:09:22.288516 11070 consensus_queue.cc:1035] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:09:22.289719 11070 raft_consensus.cc:2932] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 9d7c8c0f7907428eab22633702ead7ab (127.7.39.134) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } }
I20240501 14:09:22.290163 9215 raft_consensus.cc:2932] T 3137e7904ad04fc79d4028c1882e2412 P b4105a6d531d401faadf67b894fec79d [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 9d7c8c0f7907428eab22633702ead7ab (127.7.39.134) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } }
I20240501 14:09:22.292248 7642 raft_consensus.cc:2932] T 3137e7904ad04fc79d4028c1882e2412 P 9e44efa72e7d4421b284ae74d1cb9634 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 9d7c8c0f7907428eab22633702ead7ab (127.7.39.134) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } }
I20240501 14:09:22.292520 7364 catalog_manager.cc:5484] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 reported cstate change: config changed from index 9 to 10, 9d7c8c0f7907428eab22633702ead7ab (127.7.39.134) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "47eabc135e9f4237b036097085e729d8" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:22.292809 9083 raft_consensus.cc:2932] T 3137e7904ad04fc79d4028c1882e2412 P 9d7c8c0f7907428eab22633702ead7ab [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 9d7c8c0f7907428eab22633702ead7ab (127.7.39.134) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } }
I20240501 14:09:22.299474 7904 consensus_queue.cc:237] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } }
I20240501 14:09:22.300745 9215 raft_consensus.cc:1261] T 3137e7904ad04fc79d4028c1882e2412 P b4105a6d531d401faadf67b894fec79d [term 2 FOLLOWER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 14:09:22.300742 9083 raft_consensus.cc:1261] T 3137e7904ad04fc79d4028c1882e2412 P 9d7c8c0f7907428eab22633702ead7ab [term 2 FOLLOWER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 14:09:22.301120 11090 consensus_queue.cc:1035] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:09:22.301306 11088 consensus_queue.cc:1035] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:09:22.302974 11070 raft_consensus.cc:2932] T 3137e7904ad04fc79d4028c1882e2412 P 47eabc135e9f4237b036097085e729d8 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 9e44efa72e7d4421b284ae74d1cb9634 (127.7.39.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } }
I20240501 14:09:22.303402 9083 raft_consensus.cc:2932] T 3137e7904ad04fc79d4028c1882e2412 P 9d7c8c0f7907428eab22633702ead7ab [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 9e44efa72e7d4421b284ae74d1cb9634 (127.7.39.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } }
I20240501 14:09:22.304859 9215 raft_consensus.cc:2932] T 3137e7904ad04fc79d4028c1882e2412 P b4105a6d531d401faadf67b894fec79d [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 9e44efa72e7d4421b284ae74d1cb9634 (127.7.39.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } }
I20240501 14:09:22.305147 7354 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3137e7904ad04fc79d4028c1882e2412 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:22.305730 7365 catalog_manager.cc:5484] T 3137e7904ad04fc79d4028c1882e2412 P 9d7c8c0f7907428eab22633702ead7ab reported cstate change: config changed from index 10 to 11, VOTER 9e44efa72e7d4421b284ae74d1cb9634 (127.7.39.130) evicted. New cstate: current_term: 2 leader_uuid: "47eabc135e9f4237b036097085e729d8" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } }
I20240501 14:09:22.308377 7620 tablet_service.cc:1514] Processing DeleteTablet for tablet 3137e7904ad04fc79d4028c1882e2412 with delete_type TABLET_DATA_TOMBSTONED (TS 9e44efa72e7d4421b284ae74d1cb9634 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:60602
I20240501 14:09:22.308774 11104 tablet_replica.cc:330] T 3137e7904ad04fc79d4028c1882e2412 P 9e44efa72e7d4421b284ae74d1cb9634: stopping tablet replica
I20240501 14:09:22.309065 11104 raft_consensus.cc:2229] T 3137e7904ad04fc79d4028c1882e2412 P 9e44efa72e7d4421b284ae74d1cb9634 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:22.309296 11104 raft_consensus.cc:2258] T 3137e7904ad04fc79d4028c1882e2412 P 9e44efa72e7d4421b284ae74d1cb9634 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:22.309993 11104 ts_tablet_manager.cc:1875] T 3137e7904ad04fc79d4028c1882e2412 P 9e44efa72e7d4421b284ae74d1cb9634: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:22.311944 11104 ts_tablet_manager.cc:1888] T 3137e7904ad04fc79d4028c1882e2412 P 9e44efa72e7d4421b284ae74d1cb9634: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240501 14:09:22.312011 11104 log.cc:1196] T 3137e7904ad04fc79d4028c1882e2412 P 9e44efa72e7d4421b284ae74d1cb9634: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal/wals/3137e7904ad04fc79d4028c1882e2412
I20240501 14:09:22.312485 7355 catalog_manager.cc:4830] TS 9e44efa72e7d4421b284ae74d1cb9634 (127.7.39.130:46433): tablet 3137e7904ad04fc79d4028c1882e2412 (table rebalance_test_table_2 [id=4363cbae8a774bc9b70e97f51629b7b3]) successfully deleted
I20240501 14:09:22.422324 11088 raft_consensus.cc:1050] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8: attempting to promote NON_VOTER 9faf312806f54f2e8a4a807b234e0d1b to VOTER
I20240501 14:09:22.422734 11088 consensus_queue.cc:237] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } }
I20240501 14:09:22.424152 8953 raft_consensus.cc:1261] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9faf312806f54f2e8a4a807b234e0d1b [term 1 LEARNER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:22.424160 7771 raft_consensus.cc:1261] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:22.424203 9215 raft_consensus.cc:1261] T 2112bf96dc3343678aa6f46bf2a3b1f2 P b4105a6d531d401faadf67b894fec79d [term 1 FOLLOWER]: Refusing update from remote peer 47eabc135e9f4237b036097085e729d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:22.424571 11090 consensus_queue.cc:1035] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:22.424743 10954 consensus_queue.cc:1035] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:22.424979 11070 consensus_queue.cc:1035] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:22.427115 11090 raft_consensus.cc:2932] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9faf312806f54f2e8a4a807b234e0d1b (127.7.39.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } }
I20240501 14:09:22.427461 7771 raft_consensus.cc:2932] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9faf312806f54f2e8a4a807b234e0d1b (127.7.39.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } }
I20240501 14:09:22.427454 9215 raft_consensus.cc:2932] T 2112bf96dc3343678aa6f46bf2a3b1f2 P b4105a6d531d401faadf67b894fec79d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9faf312806f54f2e8a4a807b234e0d1b (127.7.39.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } }
I20240501 14:09:22.429776 8953 raft_consensus.cc:2932] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9faf312806f54f2e8a4a807b234e0d1b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9faf312806f54f2e8a4a807b234e0d1b (127.7.39.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } }
I20240501 14:09:22.429888 7366 catalog_manager.cc:5484] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9ae5a87ca07b40a788a42734b24f66c6 reported cstate change: config changed from index 8 to 9, 9faf312806f54f2e8a4a807b234e0d1b (127.7.39.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "47eabc135e9f4237b036097085e729d8" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } }
W20240501 14:09:22.444898 7839 consensus_peers.cc:485] T 4d282d79e1664f7ea8912e0c5166eec3 P 47eabc135e9f4237b036097085e729d8 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:09:22.506176 7708 consensus_peers.cc:485] T dcf673c75a8b48da95806e1c43309dda P 9ae5a87ca07b40a788a42734b24f66c6 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 14:09:22.507205 7904 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2112bf96dc3343678aa6f46bf2a3b1f2"
dest_uuid: "47eabc135e9f4237b036097085e729d8"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53782
I20240501 14:09:22.507359 7904 raft_consensus.cc:595] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:22.520224 7771 consensus_queue.cc:237] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: NON_VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: true } }
I20240501 14:09:22.521741 7642 raft_consensus.cc:1261] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 FOLLOWER]: Refusing update from remote peer 9ae5a87ca07b40a788a42734b24f66c6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:22.521806 7904 raft_consensus.cc:1261] T 6841e1b32b86454f9e40b64c141a2a6b P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Refusing update from remote peer 9ae5a87ca07b40a788a42734b24f66c6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:09:22.522083 7709 consensus_peers.cc:485] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6 -> Peer 9d7c8c0f7907428eab22633702ead7ab (127.7.39.134:38699): Couldn't send request to peer 9d7c8c0f7907428eab22633702ead7ab. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6841e1b32b86454f9e40b64c141a2a6b. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:22.522198 11132 consensus_queue.cc:1035] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:22.522395 10813 consensus_queue.cc:1035] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:22.524263 11132 raft_consensus.cc:2932] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9d7c8c0f7907428eab22633702ead7ab (127.7.39.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: NON_VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: true } } }
I20240501 14:09:22.524650 7904 raft_consensus.cc:2932] T 6841e1b32b86454f9e40b64c141a2a6b P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9d7c8c0f7907428eab22633702ead7ab (127.7.39.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: NON_VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: true } } }
I20240501 14:09:22.524777 7642 raft_consensus.cc:2932] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9d7c8c0f7907428eab22633702ead7ab (127.7.39.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: NON_VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: true } } }
I20240501 14:09:22.527592 7367 catalog_manager.cc:5484] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634 reported cstate change: config changed from index -1 to 4, NON_VOTER 9d7c8c0f7907428eab22633702ead7ab (127.7.39.134) added. New cstate: current_term: 1 leader_uuid: "9ae5a87ca07b40a788a42734b24f66c6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: NON_VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: true } } }
I20240501 14:09:22.543489 7904 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2112bf96dc3343678aa6f46bf2a3b1f2"
dest_uuid: "47eabc135e9f4237b036097085e729d8"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53810
I20240501 14:09:22.543614 7904 raft_consensus.cc:595] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:22.760437 7904 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2112bf96dc3343678aa6f46bf2a3b1f2"
dest_uuid: "47eabc135e9f4237b036097085e729d8"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53838
I20240501 14:09:22.760581 7904 raft_consensus.cc:595] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [term 1 LEADER]: Received request to transfer leadership
W20240501 14:09:22.844660 7708 consensus_peers.cc:485] T ceee52d679bf4d8e8896a6c0dde0519b P 9ae5a87ca07b40a788a42734b24f66c6 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 14:09:22.871802 10954 raft_consensus.cc:979] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8: : Instructing follower 9ae5a87ca07b40a788a42734b24f66c6 to start an election
I20240501 14:09:22.871955 10954 raft_consensus.cc:1067] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [term 1 LEADER]: Signalling peer 9ae5a87ca07b40a788a42734b24f66c6 to start an election
I20240501 14:09:22.872452 7771 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2112bf96dc3343678aa6f46bf2a3b1f2"
dest_uuid: "9ae5a87ca07b40a788a42734b24f66c6"
from {username='slave'} at 127.7.39.132:39797
I20240501 14:09:22.872606 7771 raft_consensus.cc:482] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:22.872675 7771 raft_consensus.cc:3037] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:22.873543 7771 raft_consensus.cc:504] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9ae5a87ca07b40a788a42734b24f66c6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } }
I20240501 14:09:22.874100 7771 leader_election.cc:288] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9ae5a87ca07b40a788a42734b24f66c6 [CANDIDATE]: Term 2 election: Requested vote from peers 47eabc135e9f4237b036097085e729d8 (127.7.39.132:42369), b4105a6d531d401faadf67b894fec79d (127.7.39.135:39611), 9faf312806f54f2e8a4a807b234e0d1b (127.7.39.133:38805)
I20240501 14:09:22.874733 7904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2112bf96dc3343678aa6f46bf2a3b1f2" candidate_uuid: "9ae5a87ca07b40a788a42734b24f66c6" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "47eabc135e9f4237b036097085e729d8"
I20240501 14:09:22.874881 7904 raft_consensus.cc:3032] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:09:22.874933 7904 raft_consensus.cc:729] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 47eabc135e9f4237b036097085e729d8, State: Running, Role: LEADER
I20240501 14:09:22.875160 7904 consensus_queue.cc:260] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } }
I20240501 14:09:22.875412 7904 raft_consensus.cc:3037] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:22.876417 7904 raft_consensus.cc:2454] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ae5a87ca07b40a788a42734b24f66c6 in term 2.
I20240501 14:09:22.878559 8953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2112bf96dc3343678aa6f46bf2a3b1f2" candidate_uuid: "9ae5a87ca07b40a788a42734b24f66c6" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "9faf312806f54f2e8a4a807b234e0d1b"
I20240501 14:09:22.878583 9215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2112bf96dc3343678aa6f46bf2a3b1f2" candidate_uuid: "9ae5a87ca07b40a788a42734b24f66c6" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "b4105a6d531d401faadf67b894fec79d"
I20240501 14:09:22.878749 9215 raft_consensus.cc:3037] T 2112bf96dc3343678aa6f46bf2a3b1f2 P b4105a6d531d401faadf67b894fec79d [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:22.878736 8953 raft_consensus.cc:3037] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9faf312806f54f2e8a4a807b234e0d1b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:22.879632 8953 raft_consensus.cc:2454] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9faf312806f54f2e8a4a807b234e0d1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ae5a87ca07b40a788a42734b24f66c6 in term 2.
I20240501 14:09:22.879637 9215 raft_consensus.cc:2454] T 2112bf96dc3343678aa6f46bf2a3b1f2 P b4105a6d531d401faadf67b894fec79d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ae5a87ca07b40a788a42734b24f66c6 in term 2.
I20240501 14:09:22.879972 7707 leader_election.cc:302] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9ae5a87ca07b40a788a42734b24f66c6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 47eabc135e9f4237b036097085e729d8, 9ae5a87ca07b40a788a42734b24f66c6, 9faf312806f54f2e8a4a807b234e0d1b; no voters:
I20240501 14:09:22.880163 11135 raft_consensus.cc:2781] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9ae5a87ca07b40a788a42734b24f66c6 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:22.880254 11135 raft_consensus.cc:686] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9ae5a87ca07b40a788a42734b24f66c6 [term 2 LEADER]: Becoming Leader. State: Replica: 9ae5a87ca07b40a788a42734b24f66c6, State: Running, Role: LEADER
I20240501 14:09:22.880434 11135 consensus_queue.cc:237] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9ae5a87ca07b40a788a42734b24f66c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } }
I20240501 14:09:22.882233 7364 catalog_manager.cc:5484] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9ae5a87ca07b40a788a42734b24f66c6 reported cstate change: term changed from 1 to 2, leader changed from 47eabc135e9f4237b036097085e729d8 (127.7.39.132) to 9ae5a87ca07b40a788a42734b24f66c6 (127.7.39.131). New cstate: current_term: 2 leader_uuid: "9ae5a87ca07b40a788a42734b24f66c6" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:23.077891 11181 ts_tablet_manager.cc:902] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab: Initiating tablet copy from peer 9ae5a87ca07b40a788a42734b24f66c6 (127.7.39.131:34697)
I20240501 14:09:23.078362 11181 tablet_copy_client.cc:308] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.131:34697
I20240501 14:09:23.081748 7793 tablet_copy_service.cc:140] P 9ae5a87ca07b40a788a42734b24f66c6: Received BeginTabletCopySession request for tablet 6841e1b32b86454f9e40b64c141a2a6b from peer 9d7c8c0f7907428eab22633702ead7ab ({username='slave'} at 127.7.39.134:51457)
I20240501 14:09:23.081881 7793 tablet_copy_service.cc:161] P 9ae5a87ca07b40a788a42734b24f66c6: Beginning new tablet copy session on tablet 6841e1b32b86454f9e40b64c141a2a6b from peer 9d7c8c0f7907428eab22633702ead7ab at {username='slave'} at 127.7.39.134:51457: session id = 9d7c8c0f7907428eab22633702ead7ab-6841e1b32b86454f9e40b64c141a2a6b
I20240501 14:09:23.083187 7793 tablet_copy_source_session.cc:205] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:23.084344 11181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6841e1b32b86454f9e40b64c141a2a6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:23.086084 11181 tablet_copy_client.cc:783] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:23.086261 11181 tablet_copy_client.cc:647] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:23.087646 11185 tablet_copy_client.cc:1002] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab: tablet copy: Injecting latency into file download: 1500
I20240501 14:09:23.260221 7904 raft_consensus.cc:1261] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [term 2 FOLLOWER]: Refusing update from remote peer 9ae5a87ca07b40a788a42734b24f66c6: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 14:09:23.260802 11135 consensus_queue.cc:1035] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9ae5a87ca07b40a788a42734b24f66c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:09:23.263140 8953 raft_consensus.cc:1261] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9faf312806f54f2e8a4a807b234e0d1b [term 2 FOLLOWER]: Refusing update from remote peer 9ae5a87ca07b40a788a42734b24f66c6: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 14:09:23.263603 10813 consensus_queue.cc:1035] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9ae5a87ca07b40a788a42734b24f66c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:09:23.266167 9215 raft_consensus.cc:1261] T 2112bf96dc3343678aa6f46bf2a3b1f2 P b4105a6d531d401faadf67b894fec79d [term 2 FOLLOWER]: Refusing update from remote peer 9ae5a87ca07b40a788a42734b24f66c6: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 14:09:23.266551 10813 consensus_queue.cc:1035] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9ae5a87ca07b40a788a42734b24f66c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240501 14:09:23.431602 7708 consensus_peers.cc:485] T 2880045de45743bdb5358a17d93b2245 P 9ae5a87ca07b40a788a42734b24f66c6 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 14:09:23.841512 7771 consensus_queue.cc:237] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9ae5a87ca07b40a788a42734b24f66c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } }
I20240501 14:09:23.842862 8953 raft_consensus.cc:1261] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9faf312806f54f2e8a4a807b234e0d1b [term 2 FOLLOWER]: Refusing update from remote peer 9ae5a87ca07b40a788a42734b24f66c6: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 14:09:23.843258 9215 raft_consensus.cc:1261] T 2112bf96dc3343678aa6f46bf2a3b1f2 P b4105a6d531d401faadf67b894fec79d [term 2 FOLLOWER]: Refusing update from remote peer 9ae5a87ca07b40a788a42734b24f66c6: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 14:09:23.843365 11199 consensus_queue.cc:1035] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9ae5a87ca07b40a788a42734b24f66c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:09:23.843685 10813 consensus_queue.cc:1035] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9ae5a87ca07b40a788a42734b24f66c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:09:23.845276 11135 raft_consensus.cc:2932] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9ae5a87ca07b40a788a42734b24f66c6 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 47eabc135e9f4237b036097085e729d8 (127.7.39.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } }
I20240501 14:09:23.845775 9215 raft_consensus.cc:2932] T 2112bf96dc3343678aa6f46bf2a3b1f2 P b4105a6d531d401faadf67b894fec79d [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 47eabc135e9f4237b036097085e729d8 (127.7.39.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } }
I20240501 14:09:23.845875 8953 raft_consensus.cc:2932] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9faf312806f54f2e8a4a807b234e0d1b [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 47eabc135e9f4237b036097085e729d8 (127.7.39.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } }
I20240501 14:09:23.847440 7355 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2112bf96dc3343678aa6f46bf2a3b1f2 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:23.848218 7364 catalog_manager.cc:5484] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 9ae5a87ca07b40a788a42734b24f66c6 reported cstate change: config changed from index 9 to 11, VOTER 47eabc135e9f4237b036097085e729d8 (127.7.39.132) evicted. New cstate: current_term: 2 leader_uuid: "9ae5a87ca07b40a788a42734b24f66c6" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:23.851106 7882 tablet_service.cc:1514] Processing DeleteTablet for tablet 2112bf96dc3343678aa6f46bf2a3b1f2 with delete_type TABLET_DATA_TOMBSTONED (TS 47eabc135e9f4237b036097085e729d8 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:43764
I20240501 14:09:23.851629 11240 tablet_replica.cc:330] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8: stopping tablet replica
I20240501 14:09:23.851835 11240 raft_consensus.cc:2229] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:23.852002 11240 raft_consensus.cc:2258] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:23.852742 11240 ts_tablet_manager.cc:1875] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:23.854799 11240 ts_tablet_manager.cc:1888] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240501 14:09:23.854892 11240 log.cc:1196] T 2112bf96dc3343678aa6f46bf2a3b1f2 P 47eabc135e9f4237b036097085e729d8: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal/wals/2112bf96dc3343678aa6f46bf2a3b1f2
I20240501 14:09:23.855453 7354 catalog_manager.cc:4830] TS 47eabc135e9f4237b036097085e729d8 (127.7.39.132:42369): tablet 2112bf96dc3343678aa6f46bf2a3b1f2 (table rebalance_test_table_2 [id=4363cbae8a774bc9b70e97f51629b7b3]) successfully deleted
I20240501 14:09:24.045717 7642 consensus_queue.cc:237] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: NON_VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: true } }
W20240501 14:09:24.047434 7576 consensus_peers.cc:485] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 -> Peer 9d32e843754e44d0b9aae6f472a4d9a2 (127.7.39.136:37475): Couldn't send request to peer 9d32e843754e44d0b9aae6f472a4d9a2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ab7c1f81f7524d55bd43065890088eaf. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:24.047408 7904 raft_consensus.cc:1261] T ab7c1f81f7524d55bd43065890088eaf P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Refusing update from remote peer 9e44efa72e7d4421b284ae74d1cb9634: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:24.047468 8953 raft_consensus.cc:1261] T ab7c1f81f7524d55bd43065890088eaf P 9faf312806f54f2e8a4a807b234e0d1b [term 1 FOLLOWER]: Refusing update from remote peer 9e44efa72e7d4421b284ae74d1cb9634: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:24.047848 11259 consensus_queue.cc:1035] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:24.048727 11002 consensus_queue.cc:1035] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:24.050789 11258 raft_consensus.cc:2932] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9d32e843754e44d0b9aae6f472a4d9a2 (127.7.39.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: NON_VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: true } } }
I20240501 14:09:24.051410 7904 raft_consensus.cc:2932] T ab7c1f81f7524d55bd43065890088eaf P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9d32e843754e44d0b9aae6f472a4d9a2 (127.7.39.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: NON_VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: true } } }
I20240501 14:09:24.051415 8953 raft_consensus.cc:2932] T ab7c1f81f7524d55bd43065890088eaf P 9faf312806f54f2e8a4a807b234e0d1b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9d32e843754e44d0b9aae6f472a4d9a2 (127.7.39.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: NON_VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: true } } }
I20240501 14:09:24.054217 7367 catalog_manager.cc:5484] T ab7c1f81f7524d55bd43065890088eaf P 9faf312806f54f2e8a4a807b234e0d1b reported cstate change: config changed from index 7 to 8, NON_VOTER 9d32e843754e44d0b9aae6f472a4d9a2 (127.7.39.136) added. New cstate: current_term: 1 leader_uuid: "9e44efa72e7d4421b284ae74d1cb9634" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: NON_VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: true } } }
W20240501 14:09:24.303674 7577 consensus_peers.cc:485] T 2cf2f85819584fdf872f6c590d11f873 P 9e44efa72e7d4421b284ae74d1cb9634 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:09:24.436746 7577 consensus_peers.cc:485] T df856440b07b418394477cf368b765c1 P 9e44efa72e7d4421b284ae74d1cb9634 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 14:09:24.512213 11292 ts_tablet_manager.cc:902] T ab7c1f81f7524d55bd43065890088eaf P 9d32e843754e44d0b9aae6f472a4d9a2: Initiating tablet copy from peer 9e44efa72e7d4421b284ae74d1cb9634 (127.7.39.130:46433)
I20240501 14:09:24.512691 11292 tablet_copy_client.cc:308] T ab7c1f81f7524d55bd43065890088eaf P 9d32e843754e44d0b9aae6f472a4d9a2: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.130:46433
I20240501 14:09:24.513180 7662 tablet_copy_service.cc:140] P 9e44efa72e7d4421b284ae74d1cb9634: Received BeginTabletCopySession request for tablet ab7c1f81f7524d55bd43065890088eaf from peer 9d32e843754e44d0b9aae6f472a4d9a2 ({username='slave'} at 127.7.39.136:48105)
I20240501 14:09:24.513286 7662 tablet_copy_service.cc:161] P 9e44efa72e7d4421b284ae74d1cb9634: Beginning new tablet copy session on tablet ab7c1f81f7524d55bd43065890088eaf from peer 9d32e843754e44d0b9aae6f472a4d9a2 at {username='slave'} at 127.7.39.136:48105: session id = 9d32e843754e44d0b9aae6f472a4d9a2-ab7c1f81f7524d55bd43065890088eaf
I20240501 14:09:24.514042 7662 tablet_copy_source_session.cc:205] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:24.514886 11292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab7c1f81f7524d55bd43065890088eaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:24.516630 11292 tablet_copy_client.cc:783] T ab7c1f81f7524d55bd43065890088eaf P 9d32e843754e44d0b9aae6f472a4d9a2: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:24.516789 11292 tablet_copy_client.cc:647] T ab7c1f81f7524d55bd43065890088eaf P 9d32e843754e44d0b9aae6f472a4d9a2: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:24.517773 11296 tablet_copy_client.cc:1002] T ab7c1f81f7524d55bd43065890088eaf P 9d32e843754e44d0b9aae6f472a4d9a2: tablet copy: Injecting latency into file download: 1500
I20240501 14:09:24.588039 11181 tablet_copy_client.cc:523] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:24.589398 11181 tablet_bootstrap.cc:492] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab: Bootstrap starting.
I20240501 14:09:24.606446 11181 tablet_bootstrap.cc:492] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:24.606714 11181 tablet_bootstrap.cc:492] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab: Bootstrap complete.
I20240501 14:09:24.606873 11181 ts_tablet_manager.cc:1367] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.000s
I20240501 14:09:24.607277 11181 raft_consensus.cc:348] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: NON_VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: true } }
I20240501 14:09:24.607388 11181 raft_consensus.cc:729] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9d7c8c0f7907428eab22633702ead7ab, State: Initialized, Role: LEARNER
I20240501 14:09:24.607460 11181 consensus_queue.cc:260] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: NON_VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: true } }
I20240501 14:09:24.607719 11181 ts_tablet_manager.cc:1398] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab: Time spent starting tablet: real 0.001s user 0.005s sys 0.000s
I20240501 14:09:24.608448 7793 tablet_copy_service.cc:342] P 9ae5a87ca07b40a788a42734b24f66c6: Request end of tablet copy session 9d7c8c0f7907428eab22633702ead7ab-6841e1b32b86454f9e40b64c141a2a6b received from {username='slave'} at 127.7.39.134:51457
I20240501 14:09:24.608572 7793 tablet_copy_service.cc:433] P 9ae5a87ca07b40a788a42734b24f66c6: ending tablet copy session 9d7c8c0f7907428eab22633702ead7ab-6841e1b32b86454f9e40b64c141a2a6b on tablet 6841e1b32b86454f9e40b64c141a2a6b with peer 9d7c8c0f7907428eab22633702ead7ab
W20240501 14:09:24.677405 7839 consensus_peers.cc:485] T 14e325d8d3184c9192b8cebafb414190 P 47eabc135e9f4237b036097085e729d8 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:09:24.945792 7708 consensus_peers.cc:485] T dcf673c75a8b48da95806e1c43309dda P 9ae5a87ca07b40a788a42734b24f66c6 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 14:09:24.955871 9083 raft_consensus.cc:1203] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240501 14:09:25.192273 7839 consensus_peers.cc:485] T 4d282d79e1664f7ea8912e0c5166eec3 P 47eabc135e9f4237b036097085e729d8 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:09:25.279129 7355 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111) [suppressed 4 similar messages]
W20240501 14:09:25.279898 7355 catalog_manager.cc:4573] TS ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet ae06bed9bd844a05ae2ac69148bf5bad: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111)
W20240501 14:09:25.351714 7577 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 14:09:25.435407 7708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:09:25.436105 7708 consensus_peers.cc:485] T ceee52d679bf4d8e8896a6c0dde0519b P 9ae5a87ca07b40a788a42734b24f66c6 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 14:09:25.537873 11346 raft_consensus.cc:1050] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6: attempting to promote NON_VOTER 9d7c8c0f7907428eab22633702ead7ab to VOTER
I20240501 14:09:25.538370 11346 consensus_queue.cc:237] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } }
I20240501 14:09:25.539688 9083 raft_consensus.cc:1261] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab [term 1 LEARNER]: Refusing update from remote peer 9ae5a87ca07b40a788a42734b24f66c6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:25.539801 7904 raft_consensus.cc:1261] T 6841e1b32b86454f9e40b64c141a2a6b P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Refusing update from remote peer 9ae5a87ca07b40a788a42734b24f66c6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:25.540201 7642 raft_consensus.cc:1261] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 FOLLOWER]: Refusing update from remote peer 9ae5a87ca07b40a788a42734b24f66c6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:25.540179 11346 consensus_queue.cc:1035] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:25.540397 11348 consensus_queue.cc:1035] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:25.540637 11345 consensus_queue.cc:1035] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:25.542578 11348 raft_consensus.cc:2932] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9d7c8c0f7907428eab22633702ead7ab (127.7.39.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } }
I20240501 14:09:25.543006 9083 raft_consensus.cc:2932] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9d7c8c0f7907428eab22633702ead7ab (127.7.39.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } }
I20240501 14:09:25.543192 7642 raft_consensus.cc:2932] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9d7c8c0f7907428eab22633702ead7ab (127.7.39.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } }
I20240501 14:09:25.544720 7904 raft_consensus.cc:2932] T 6841e1b32b86454f9e40b64c141a2a6b P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9d7c8c0f7907428eab22633702ead7ab (127.7.39.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } }
I20240501 14:09:25.545984 7364 catalog_manager.cc:5484] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab reported cstate change: config changed from index 4 to 5, 9d7c8c0f7907428eab22633702ead7ab (127.7.39.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9ae5a87ca07b40a788a42734b24f66c6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } }
I20240501 14:09:25.562144 7771 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6841e1b32b86454f9e40b64c141a2a6b"
dest_uuid: "9ae5a87ca07b40a788a42734b24f66c6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35412
I20240501 14:09:25.562362 7771 raft_consensus.cc:595] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 LEADER]: Received request to transfer leadership
W20240501 14:09:25.583036 7839 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 14:09:25.779775 7771 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6841e1b32b86454f9e40b64c141a2a6b"
dest_uuid: "9ae5a87ca07b40a788a42734b24f66c6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35426
I20240501 14:09:25.779929 7771 raft_consensus.cc:595] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 LEADER]: Received request to transfer leadership
W20240501 14:09:25.984589 7708 consensus_peers.cc:485] T 2880045de45743bdb5358a17d93b2245 P 9ae5a87ca07b40a788a42734b24f66c6 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 14:09:25.997674 7771 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6841e1b32b86454f9e40b64c141a2a6b"
dest_uuid: "9ae5a87ca07b40a788a42734b24f66c6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35446
I20240501 14:09:25.997833 7771 raft_consensus.cc:595] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:26.018110 11292 tablet_copy_client.cc:523] T ab7c1f81f7524d55bd43065890088eaf P 9d32e843754e44d0b9aae6f472a4d9a2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:26.019574 11292 tablet_bootstrap.cc:492] T ab7c1f81f7524d55bd43065890088eaf P 9d32e843754e44d0b9aae6f472a4d9a2: Bootstrap starting.
I20240501 14:09:26.038137 11292 tablet_bootstrap.cc:492] T ab7c1f81f7524d55bd43065890088eaf P 9d32e843754e44d0b9aae6f472a4d9a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:26.038491 11292 tablet_bootstrap.cc:492] T ab7c1f81f7524d55bd43065890088eaf P 9d32e843754e44d0b9aae6f472a4d9a2: Bootstrap complete.
I20240501 14:09:26.038681 11292 ts_tablet_manager.cc:1367] T ab7c1f81f7524d55bd43065890088eaf P 9d32e843754e44d0b9aae6f472a4d9a2: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.003s
I20240501 14:09:26.039110 11292 raft_consensus.cc:348] T ab7c1f81f7524d55bd43065890088eaf P 9d32e843754e44d0b9aae6f472a4d9a2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: NON_VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: true } }
I20240501 14:09:26.039216 11292 raft_consensus.cc:729] T ab7c1f81f7524d55bd43065890088eaf P 9d32e843754e44d0b9aae6f472a4d9a2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9d32e843754e44d0b9aae6f472a4d9a2, State: Initialized, Role: LEARNER
I20240501 14:09:26.039319 11292 consensus_queue.cc:260] T ab7c1f81f7524d55bd43065890088eaf P 9d32e843754e44d0b9aae6f472a4d9a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: NON_VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: true } }
I20240501 14:09:26.039693 11292 ts_tablet_manager.cc:1398] T ab7c1f81f7524d55bd43065890088eaf P 9d32e843754e44d0b9aae6f472a4d9a2: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 14:09:26.040351 7662 tablet_copy_service.cc:342] P 9e44efa72e7d4421b284ae74d1cb9634: Request end of tablet copy session 9d32e843754e44d0b9aae6f472a4d9a2-ab7c1f81f7524d55bd43065890088eaf received from {username='slave'} at 127.7.39.136:48105
I20240501 14:09:26.040436 7662 tablet_copy_service.cc:433] P 9e44efa72e7d4421b284ae74d1cb9634: ending tablet copy session 9d32e843754e44d0b9aae6f472a4d9a2-ab7c1f81f7524d55bd43065890088eaf on tablet ab7c1f81f7524d55bd43065890088eaf with peer 9d32e843754e44d0b9aae6f472a4d9a2
I20240501 14:09:26.043812 11348 raft_consensus.cc:979] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6: : Instructing follower 9e44efa72e7d4421b284ae74d1cb9634 to start an election
I20240501 14:09:26.043920 11348 raft_consensus.cc:1067] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 LEADER]: Signalling peer 9e44efa72e7d4421b284ae74d1cb9634 to start an election
I20240501 14:09:26.044450 7642 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6841e1b32b86454f9e40b64c141a2a6b"
dest_uuid: "9e44efa72e7d4421b284ae74d1cb9634"
from {username='slave'} at 127.7.39.131:52145
I20240501 14:09:26.044610 7642 raft_consensus.cc:482] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:26.044677 7642 raft_consensus.cc:3037] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:26.045464 7642 raft_consensus.cc:504] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } }
I20240501 14:09:26.045907 7642 leader_election.cc:288] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634 [CANDIDATE]: Term 2 election: Requested vote from peers 9ae5a87ca07b40a788a42734b24f66c6 (127.7.39.131:34697), 47eabc135e9f4237b036097085e729d8 (127.7.39.132:42369), 9d7c8c0f7907428eab22633702ead7ab (127.7.39.134:38699)
I20240501 14:09:26.046202 7904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6841e1b32b86454f9e40b64c141a2a6b" candidate_uuid: "9e44efa72e7d4421b284ae74d1cb9634" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "47eabc135e9f4237b036097085e729d8"
I20240501 14:09:26.046293 7771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6841e1b32b86454f9e40b64c141a2a6b" candidate_uuid: "9e44efa72e7d4421b284ae74d1cb9634" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "9ae5a87ca07b40a788a42734b24f66c6"
I20240501 14:09:26.046350 7904 raft_consensus.cc:3037] T 6841e1b32b86454f9e40b64c141a2a6b P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:26.046391 7771 raft_consensus.cc:3032] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:09:26.046345 9083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6841e1b32b86454f9e40b64c141a2a6b" candidate_uuid: "9e44efa72e7d4421b284ae74d1cb9634" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "9d7c8c0f7907428eab22633702ead7ab"
I20240501 14:09:26.046430 7771 raft_consensus.cc:729] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9ae5a87ca07b40a788a42734b24f66c6, State: Running, Role: LEADER
I20240501 14:09:26.046466 9083 raft_consensus.cc:3037] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:26.046587 7771 consensus_queue.cc:260] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } }
I20240501 14:09:26.046772 7771 raft_consensus.cc:3037] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:26.047241 9083 raft_consensus.cc:2454] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e44efa72e7d4421b284ae74d1cb9634 in term 2.
I20240501 14:09:26.047386 7904 raft_consensus.cc:2454] T 6841e1b32b86454f9e40b64c141a2a6b P 47eabc135e9f4237b036097085e729d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e44efa72e7d4421b284ae74d1cb9634 in term 2.
I20240501 14:09:26.047737 7576 leader_election.cc:302] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 47eabc135e9f4237b036097085e729d8, 9d7c8c0f7907428eab22633702ead7ab, 9e44efa72e7d4421b284ae74d1cb9634; no voters:
I20240501 14:09:26.047761 7771 raft_consensus.cc:2454] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e44efa72e7d4421b284ae74d1cb9634 in term 2.
I20240501 14:09:26.047899 11315 raft_consensus.cc:2781] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:26.047992 11315 raft_consensus.cc:686] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634 [term 2 LEADER]: Becoming Leader. State: Replica: 9e44efa72e7d4421b284ae74d1cb9634, State: Running, Role: LEADER
I20240501 14:09:26.048157 11315 consensus_queue.cc:237] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } }
I20240501 14:09:26.049928 7365 catalog_manager.cc:5484] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634 reported cstate change: term changed from 1 to 2, leader changed from 9ae5a87ca07b40a788a42734b24f66c6 (127.7.39.131) to 9e44efa72e7d4421b284ae74d1cb9634 (127.7.39.130). New cstate: current_term: 2 leader_uuid: "9e44efa72e7d4421b284ae74d1cb9634" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:26.356791 9346 raft_consensus.cc:1203] T ab7c1f81f7524d55bd43065890088eaf P 9d32e843754e44d0b9aae6f472a4d9a2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240501 14:09:26.427714 7904 raft_consensus.cc:1261] T 6841e1b32b86454f9e40b64c141a2a6b P 47eabc135e9f4237b036097085e729d8 [term 2 FOLLOWER]: Refusing update from remote peer 9e44efa72e7d4421b284ae74d1cb9634: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:09:26.428112 11315 consensus_queue.cc:1035] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:26.560132 7771 raft_consensus.cc:1261] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6 [term 2 FOLLOWER]: Refusing update from remote peer 9e44efa72e7d4421b284ae74d1cb9634: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:09:26.560510 11315 consensus_queue.cc:1035] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:26.564307 9083 raft_consensus.cc:1261] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab [term 2 FOLLOWER]: Refusing update from remote peer 9e44efa72e7d4421b284ae74d1cb9634: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:09:26.564874 11315 consensus_queue.cc:1035] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 14:09:26.712227 7577 consensus_peers.cc:485] T 2cf2f85819584fdf872f6c590d11f873 P 9e44efa72e7d4421b284ae74d1cb9634 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 14:09:26.768379 11431 raft_consensus.cc:1050] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634: attempting to promote NON_VOTER 9d32e843754e44d0b9aae6f472a4d9a2 to VOTER
I20240501 14:09:26.768898 11431 consensus_queue.cc:237] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } }
I20240501 14:09:26.770257 9346 raft_consensus.cc:1261] T ab7c1f81f7524d55bd43065890088eaf P 9d32e843754e44d0b9aae6f472a4d9a2 [term 1 LEARNER]: Refusing update from remote peer 9e44efa72e7d4421b284ae74d1cb9634: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:26.770398 8953 raft_consensus.cc:1261] T ab7c1f81f7524d55bd43065890088eaf P 9faf312806f54f2e8a4a807b234e0d1b [term 1 FOLLOWER]: Refusing update from remote peer 9e44efa72e7d4421b284ae74d1cb9634: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:26.770453 7904 raft_consensus.cc:1261] T ab7c1f81f7524d55bd43065890088eaf P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Refusing update from remote peer 9e44efa72e7d4421b284ae74d1cb9634: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:26.770699 11315 consensus_queue.cc:1035] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:26.770859 11431 consensus_queue.cc:1035] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:26.771003 11448 consensus_queue.cc:1035] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:26.772841 11448 raft_consensus.cc:2932] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9d32e843754e44d0b9aae6f472a4d9a2 (127.7.39.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } }
I20240501 14:09:26.773322 8953 raft_consensus.cc:2932] T ab7c1f81f7524d55bd43065890088eaf P 9faf312806f54f2e8a4a807b234e0d1b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9d32e843754e44d0b9aae6f472a4d9a2 (127.7.39.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } }
I20240501 14:09:26.773651 9346 raft_consensus.cc:2932] T ab7c1f81f7524d55bd43065890088eaf P 9d32e843754e44d0b9aae6f472a4d9a2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9d32e843754e44d0b9aae6f472a4d9a2 (127.7.39.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } }
I20240501 14:09:26.775436 7904 raft_consensus.cc:2932] T ab7c1f81f7524d55bd43065890088eaf P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9d32e843754e44d0b9aae6f472a4d9a2 (127.7.39.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } }
I20240501 14:09:26.776223 7366 catalog_manager.cc:5484] T ab7c1f81f7524d55bd43065890088eaf P 9d32e843754e44d0b9aae6f472a4d9a2 reported cstate change: config changed from index 8 to 9, 9d32e843754e44d0b9aae6f472a4d9a2 (127.7.39.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9e44efa72e7d4421b284ae74d1cb9634" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } }
I20240501 14:09:26.780083 7642 consensus_queue.cc:237] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } }
I20240501 14:09:26.781163 8953 raft_consensus.cc:1261] T ab7c1f81f7524d55bd43065890088eaf P 9faf312806f54f2e8a4a807b234e0d1b [term 1 FOLLOWER]: Refusing update from remote peer 9e44efa72e7d4421b284ae74d1cb9634: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:09:26.781180 9346 raft_consensus.cc:1261] T ab7c1f81f7524d55bd43065890088eaf P 9d32e843754e44d0b9aae6f472a4d9a2 [term 1 FOLLOWER]: Refusing update from remote peer 9e44efa72e7d4421b284ae74d1cb9634: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:09:26.781585 11431 consensus_queue.cc:1035] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:09:26.782042 11448 consensus_queue.cc:1035] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:09:26.783247 11431 raft_consensus.cc:2932] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 47eabc135e9f4237b036097085e729d8 (127.7.39.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } }
I20240501 14:09:26.783849 9346 raft_consensus.cc:2932] T ab7c1f81f7524d55bd43065890088eaf P 9d32e843754e44d0b9aae6f472a4d9a2 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 47eabc135e9f4237b036097085e729d8 (127.7.39.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } }
I20240501 14:09:26.785046 7355 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ab7c1f81f7524d55bd43065890088eaf with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:26.785984 8953 raft_consensus.cc:2932] T ab7c1f81f7524d55bd43065890088eaf P 9faf312806f54f2e8a4a807b234e0d1b [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 47eabc135e9f4237b036097085e729d8 (127.7.39.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } }
I20240501 14:09:26.786278 7365 catalog_manager.cc:5484] T ab7c1f81f7524d55bd43065890088eaf P 9e44efa72e7d4421b284ae74d1cb9634 reported cstate change: config changed from index 9 to 10, VOTER 47eabc135e9f4237b036097085e729d8 (127.7.39.132) evicted. New cstate: current_term: 1 leader_uuid: "9e44efa72e7d4421b284ae74d1cb9634" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:26.789147 7882 tablet_service.cc:1514] Processing DeleteTablet for tablet ab7c1f81f7524d55bd43065890088eaf with delete_type TABLET_DATA_TOMBSTONED (TS 47eabc135e9f4237b036097085e729d8 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:43764
I20240501 14:09:26.789587 11456 tablet_replica.cc:330] T ab7c1f81f7524d55bd43065890088eaf P 47eabc135e9f4237b036097085e729d8: stopping tablet replica
I20240501 14:09:26.789738 11456 raft_consensus.cc:2229] T ab7c1f81f7524d55bd43065890088eaf P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:26.790006 11456 raft_consensus.cc:2258] T ab7c1f81f7524d55bd43065890088eaf P 47eabc135e9f4237b036097085e729d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:26.791152 11456 ts_tablet_manager.cc:1875] T ab7c1f81f7524d55bd43065890088eaf P 47eabc135e9f4237b036097085e729d8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:26.792997 11456 ts_tablet_manager.cc:1888] T ab7c1f81f7524d55bd43065890088eaf P 47eabc135e9f4237b036097085e729d8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 14:09:26.793062 11456 log.cc:1196] T ab7c1f81f7524d55bd43065890088eaf P 47eabc135e9f4237b036097085e729d8: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal/wals/ab7c1f81f7524d55bd43065890088eaf
I20240501 14:09:26.793576 7354 catalog_manager.cc:4830] TS 47eabc135e9f4237b036097085e729d8 (127.7.39.132:42369): tablet ab7c1f81f7524d55bd43065890088eaf (table rebalance_test_table_2 [id=4363cbae8a774bc9b70e97f51629b7b3]) successfully deleted
I20240501 14:09:27.103902 7642 consensus_queue.cc:237] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } }
I20240501 14:09:27.105118 9083 raft_consensus.cc:1261] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab [term 2 FOLLOWER]: Refusing update from remote peer 9e44efa72e7d4421b284ae74d1cb9634: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:09:27.105118 7904 raft_consensus.cc:1261] T 6841e1b32b86454f9e40b64c141a2a6b P 47eabc135e9f4237b036097085e729d8 [term 2 FOLLOWER]: Refusing update from remote peer 9e44efa72e7d4421b284ae74d1cb9634: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:09:27.105566 11431 consensus_queue.cc:1035] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:27.105777 11429 consensus_queue.cc:1035] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:27.107383 11429 raft_consensus.cc:2932] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 9ae5a87ca07b40a788a42734b24f66c6 (127.7.39.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } }
I20240501 14:09:27.107831 7904 raft_consensus.cc:2932] T 6841e1b32b86454f9e40b64c141a2a6b P 47eabc135e9f4237b036097085e729d8 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 9ae5a87ca07b40a788a42734b24f66c6 (127.7.39.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } }
I20240501 14:09:27.108120 9083 raft_consensus.cc:2932] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 9ae5a87ca07b40a788a42734b24f66c6 (127.7.39.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } }
I20240501 14:09:27.109928 7355 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6841e1b32b86454f9e40b64c141a2a6b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:27.109947 7367 catalog_manager.cc:5484] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634 reported cstate change: config changed from index 5 to 7, VOTER 9ae5a87ca07b40a788a42734b24f66c6 (127.7.39.131) evicted. New cstate: current_term: 2 leader_uuid: "9e44efa72e7d4421b284ae74d1cb9634" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:27.113341 7752 tablet_service.cc:1514] Processing DeleteTablet for tablet 6841e1b32b86454f9e40b64c141a2a6b with delete_type TABLET_DATA_TOMBSTONED (TS 9ae5a87ca07b40a788a42734b24f66c6 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:36366
I20240501 14:09:27.113798 11478 tablet_replica.cc:330] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6: stopping tablet replica
I20240501 14:09:27.113920 11478 raft_consensus.cc:2229] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:27.114010 11478 raft_consensus.cc:2258] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:27.114702 11478 ts_tablet_manager.cc:1875] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:27.116654 11478 ts_tablet_manager.cc:1888] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 14:09:27.116726 11478 log.cc:1196] T 6841e1b32b86454f9e40b64c141a2a6b P 9ae5a87ca07b40a788a42734b24f66c6: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal/wals/6841e1b32b86454f9e40b64c141a2a6b
I20240501 14:09:27.117313 7355 catalog_manager.cc:4830] TS 9ae5a87ca07b40a788a42734b24f66c6 (127.7.39.131:34697): tablet 6841e1b32b86454f9e40b64c141a2a6b (table rebalance_test_table_3 [id=1efd7e27dfbf4a88a4c2c2d4d7ebfe4f]) successfully deleted
W20240501 14:09:27.310051 7708 consensus_peers.cc:485] T dcf673c75a8b48da95806e1c43309dda P 9ae5a87ca07b40a788a42734b24f66c6 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:09:27.319849 7577 consensus_peers.cc:485] T df856440b07b418394477cf368b765c1 P 9e44efa72e7d4421b284ae74d1cb9634 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 14:09:27.327267 7882 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:27.328951 7620 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:27.330370 7752 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:27.330489 8931 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:27.339584 9195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:27.359151 9064 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:27.366245 9326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:27.393040 7642 consensus_queue.cc:237] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: true } }
I20240501 14:09:27.394373 9083 raft_consensus.cc:1261] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab [term 2 FOLLOWER]: Refusing update from remote peer 9e44efa72e7d4421b284ae74d1cb9634: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 14:09:27.394354 7904 raft_consensus.cc:1261] T 6841e1b32b86454f9e40b64c141a2a6b P 47eabc135e9f4237b036097085e729d8 [term 2 FOLLOWER]: Refusing update from remote peer 9e44efa72e7d4421b284ae74d1cb9634: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
W20240501 14:09:27.394670 7576 consensus_peers.cc:485] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634 -> Peer 9faf312806f54f2e8a4a807b234e0d1b (127.7.39.133:38805): Couldn't send request to peer 9faf312806f54f2e8a4a807b234e0d1b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6841e1b32b86454f9e40b64c141a2a6b. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:27.394748 11429 consensus_queue.cc:1035] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:27.394924 11448 consensus_queue.cc:1035] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:27.396631 11429 raft_consensus.cc:2932] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 9faf312806f54f2e8a4a807b234e0d1b (127.7.39.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: true } } }
I20240501 14:09:27.397037 9083 raft_consensus.cc:2932] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 9faf312806f54f2e8a4a807b234e0d1b (127.7.39.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: true } } }
I20240501 14:09:27.397117 7904 raft_consensus.cc:2932] T 6841e1b32b86454f9e40b64c141a2a6b P 47eabc135e9f4237b036097085e729d8 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 9faf312806f54f2e8a4a807b234e0d1b (127.7.39.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: true } } }
I20240501 14:09:27.399621 7366 catalog_manager.cc:5484] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab reported cstate change: config changed from index 7 to 8, NON_VOTER 9faf312806f54f2e8a4a807b234e0d1b (127.7.39.133) added. New cstate: current_term: 2 leader_uuid: "9e44efa72e7d4421b284ae74d1cb9634" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: true } } }
I20240501 14:09:27.407600 7771 consensus_queue.cc:237] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: NON_VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: true } }
W20240501 14:09:27.409363 7708 consensus_peers.cc:485] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 -> Peer b4105a6d531d401faadf67b894fec79d (127.7.39.135:39611): Couldn't send request to peer b4105a6d531d401faadf67b894fec79d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c747bd6504c540cebded0019d82495b4. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:27.409423 9346 raft_consensus.cc:1261] T c747bd6504c540cebded0019d82495b4 P 9d32e843754e44d0b9aae6f472a4d9a2 [term 2 FOLLOWER]: Refusing update from remote peer 9ae5a87ca07b40a788a42734b24f66c6: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 14:09:27.409426 9083 raft_consensus.cc:1261] T c747bd6504c540cebded0019d82495b4 P 9d7c8c0f7907428eab22633702ead7ab [term 2 FOLLOWER]: Refusing update from remote peer 9ae5a87ca07b40a788a42734b24f66c6: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 14:09:27.409828 11348 consensus_queue.cc:1035] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:09:27.410019 11535 consensus_queue.cc:1035] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:09:27.411900 11506 raft_consensus.cc:2932] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER b4105a6d531d401faadf67b894fec79d (127.7.39.135) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: NON_VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: true } } }
I20240501 14:09:27.412307 9346 raft_consensus.cc:2932] T c747bd6504c540cebded0019d82495b4 P 9d32e843754e44d0b9aae6f472a4d9a2 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER b4105a6d531d401faadf67b894fec79d (127.7.39.135) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: NON_VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: true } } }
I20240501 14:09:27.412439 9083 raft_consensus.cc:2932] T c747bd6504c540cebded0019d82495b4 P 9d7c8c0f7907428eab22633702ead7ab [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER b4105a6d531d401faadf67b894fec79d (127.7.39.135) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: NON_VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: true } } }
I20240501 14:09:27.414921 7364 catalog_manager.cc:5484] T c747bd6504c540cebded0019d82495b4 P 9d7c8c0f7907428eab22633702ead7ab reported cstate change: config changed from index 11 to 12, NON_VOTER b4105a6d531d401faadf67b894fec79d (127.7.39.135) added. New cstate: current_term: 2 leader_uuid: "9ae5a87ca07b40a788a42734b24f66c6" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: NON_VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: true } } }
W20240501 14:09:27.497407 7839 consensus_peers.cc:485] T 14e325d8d3184c9192b8cebafb414190 P 47eabc135e9f4237b036097085e729d8 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:09:27.602432 7839 consensus_peers.cc:485] T 4d282d79e1664f7ea8912e0c5166eec3 P 47eabc135e9f4237b036097085e729d8 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 14:09:27.884704 11569 ts_tablet_manager.cc:902] T 6841e1b32b86454f9e40b64c141a2a6b P 9faf312806f54f2e8a4a807b234e0d1b: Initiating tablet copy from peer 9e44efa72e7d4421b284ae74d1cb9634 (127.7.39.130:46433)
I20240501 14:09:27.885135 11569 tablet_copy_client.cc:308] T 6841e1b32b86454f9e40b64c141a2a6b P 9faf312806f54f2e8a4a807b234e0d1b: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.130:46433
I20240501 14:09:27.885609 7662 tablet_copy_service.cc:140] P 9e44efa72e7d4421b284ae74d1cb9634: Received BeginTabletCopySession request for tablet 6841e1b32b86454f9e40b64c141a2a6b from peer 9faf312806f54f2e8a4a807b234e0d1b ({username='slave'} at 127.7.39.133:34271)
I20240501 14:09:27.885721 7662 tablet_copy_service.cc:161] P 9e44efa72e7d4421b284ae74d1cb9634: Beginning new tablet copy session on tablet 6841e1b32b86454f9e40b64c141a2a6b from peer 9faf312806f54f2e8a4a807b234e0d1b at {username='slave'} at 127.7.39.133:34271: session id = 9faf312806f54f2e8a4a807b234e0d1b-6841e1b32b86454f9e40b64c141a2a6b
I20240501 14:09:27.886453 7662 tablet_copy_source_session.cc:205] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:27.887339 11569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6841e1b32b86454f9e40b64c141a2a6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:27.889073 11569 tablet_copy_client.cc:783] T 6841e1b32b86454f9e40b64c141a2a6b P 9faf312806f54f2e8a4a807b234e0d1b: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:27.889217 11569 tablet_copy_client.cc:647] T 6841e1b32b86454f9e40b64c141a2a6b P 9faf312806f54f2e8a4a807b234e0d1b: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:27.890240 11573 tablet_copy_client.cc:1002] T 6841e1b32b86454f9e40b64c141a2a6b P 9faf312806f54f2e8a4a807b234e0d1b: tablet copy: Injecting latency into file download: 1500
I20240501 14:09:27.989259 11574 ts_tablet_manager.cc:902] T c747bd6504c540cebded0019d82495b4 P b4105a6d531d401faadf67b894fec79d: Initiating tablet copy from peer 9ae5a87ca07b40a788a42734b24f66c6 (127.7.39.131:34697)
I20240501 14:09:27.989784 11574 tablet_copy_client.cc:308] T c747bd6504c540cebded0019d82495b4 P b4105a6d531d401faadf67b894fec79d: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.131:34697
I20240501 14:09:27.993304 7793 tablet_copy_service.cc:140] P 9ae5a87ca07b40a788a42734b24f66c6: Received BeginTabletCopySession request for tablet c747bd6504c540cebded0019d82495b4 from peer b4105a6d531d401faadf67b894fec79d ({username='slave'} at 127.7.39.135:36239)
I20240501 14:09:27.993413 7793 tablet_copy_service.cc:161] P 9ae5a87ca07b40a788a42734b24f66c6: Beginning new tablet copy session on tablet c747bd6504c540cebded0019d82495b4 from peer b4105a6d531d401faadf67b894fec79d at {username='slave'} at 127.7.39.135:36239: session id = b4105a6d531d401faadf67b894fec79d-c747bd6504c540cebded0019d82495b4
I20240501 14:09:27.994282 7793 tablet_copy_source_session.cc:205] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:27.995244 11574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c747bd6504c540cebded0019d82495b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:27.997138 11574 tablet_copy_client.cc:783] T c747bd6504c540cebded0019d82495b4 P b4105a6d531d401faadf67b894fec79d: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:27.997314 11574 tablet_copy_client.cc:647] T c747bd6504c540cebded0019d82495b4 P b4105a6d531d401faadf67b894fec79d: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:27.998662 11578 tablet_copy_client.cc:1002] T c747bd6504c540cebded0019d82495b4 P b4105a6d531d401faadf67b894fec79d: tablet copy: Injecting latency into file download: 1500
W20240501 14:09:28.124752 7355 catalog_manager.cc:4573] TS ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet ab7c1f81f7524d55bd43065890088eaf: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111)
W20240501 14:09:28.166584 7708 consensus_peers.cc:485] T ceee52d679bf4d8e8896a6c0dde0519b P 9ae5a87ca07b40a788a42734b24f66c6 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:09:28.684712 7708 consensus_peers.cc:485] T 2880045de45743bdb5358a17d93b2245 P 9ae5a87ca07b40a788a42734b24f66c6 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:09:29.110724 7577 consensus_peers.cc:485] T 2cf2f85819584fdf872f6c590d11f873 P 9e44efa72e7d4421b284ae74d1cb9634 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 14:09:29.171602 10954 consensus_queue.cc:579] T 4d282d79e1664f7ea8912e0c5166eec3 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Leader has been unable to successfully communicate with peer ae52f888f5094416a24cb3dffbff03c4 for more than 30 seconds (30.016s)
W20240501 14:09:29.366710 7577 consensus_peers.cc:485] T df856440b07b418394477cf368b765c1 P 9e44efa72e7d4421b284ae74d1cb9634 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 14:09:29.390681 11569 tablet_copy_client.cc:523] T 6841e1b32b86454f9e40b64c141a2a6b P 9faf312806f54f2e8a4a807b234e0d1b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:29.392129 11569 tablet_bootstrap.cc:492] T 6841e1b32b86454f9e40b64c141a2a6b P 9faf312806f54f2e8a4a807b234e0d1b: Bootstrap starting.
I20240501 14:09:29.418383 11569 tablet_bootstrap.cc:492] T 6841e1b32b86454f9e40b64c141a2a6b P 9faf312806f54f2e8a4a807b234e0d1b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:29.418727 11569 tablet_bootstrap.cc:492] T 6841e1b32b86454f9e40b64c141a2a6b P 9faf312806f54f2e8a4a807b234e0d1b: Bootstrap complete.
I20240501 14:09:29.418936 11569 ts_tablet_manager.cc:1367] T 6841e1b32b86454f9e40b64c141a2a6b P 9faf312806f54f2e8a4a807b234e0d1b: Time spent bootstrapping tablet: real 0.027s user 0.023s sys 0.000s
I20240501 14:09:29.419539 11569 raft_consensus.cc:348] T 6841e1b32b86454f9e40b64c141a2a6b P 9faf312806f54f2e8a4a807b234e0d1b [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: true } }
I20240501 14:09:29.419725 11569 raft_consensus.cc:729] T 6841e1b32b86454f9e40b64c141a2a6b P 9faf312806f54f2e8a4a807b234e0d1b [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 9faf312806f54f2e8a4a807b234e0d1b, State: Initialized, Role: LEARNER
I20240501 14:09:29.419890 11569 consensus_queue.cc:260] T 6841e1b32b86454f9e40b64c141a2a6b P 9faf312806f54f2e8a4a807b234e0d1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: true } }
I20240501 14:09:29.420354 11569 ts_tablet_manager.cc:1398] T 6841e1b32b86454f9e40b64c141a2a6b P 9faf312806f54f2e8a4a807b234e0d1b: Time spent starting tablet: real 0.001s user 0.005s sys 0.000s
I20240501 14:09:29.421002 7662 tablet_copy_service.cc:342] P 9e44efa72e7d4421b284ae74d1cb9634: Request end of tablet copy session 9faf312806f54f2e8a4a807b234e0d1b-6841e1b32b86454f9e40b64c141a2a6b received from {username='slave'} at 127.7.39.133:34271
I20240501 14:09:29.421100 7662 tablet_copy_service.cc:433] P 9e44efa72e7d4421b284ae74d1cb9634: ending tablet copy session 9faf312806f54f2e8a4a807b234e0d1b-6841e1b32b86454f9e40b64c141a2a6b on tablet 6841e1b32b86454f9e40b64c141a2a6b with peer 9faf312806f54f2e8a4a807b234e0d1b
W20240501 14:09:29.469158 7708 consensus_peers.cc:485] T dcf673c75a8b48da95806e1c43309dda P 9ae5a87ca07b40a788a42734b24f66c6 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 14:09:29.499024 11574 tablet_copy_client.cc:523] T c747bd6504c540cebded0019d82495b4 P b4105a6d531d401faadf67b894fec79d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:29.500530 11574 tablet_bootstrap.cc:492] T c747bd6504c540cebded0019d82495b4 P b4105a6d531d401faadf67b894fec79d: Bootstrap starting.
I20240501 14:09:29.520062 11574 tablet_bootstrap.cc:492] T c747bd6504c540cebded0019d82495b4 P b4105a6d531d401faadf67b894fec79d: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:29.520406 11574 tablet_bootstrap.cc:492] T c747bd6504c540cebded0019d82495b4 P b4105a6d531d401faadf67b894fec79d: Bootstrap complete.
I20240501 14:09:29.520583 11574 ts_tablet_manager.cc:1367] T c747bd6504c540cebded0019d82495b4 P b4105a6d531d401faadf67b894fec79d: Time spent bootstrapping tablet: real 0.020s user 0.013s sys 0.007s
I20240501 14:09:29.520964 11574 raft_consensus.cc:348] T c747bd6504c540cebded0019d82495b4 P b4105a6d531d401faadf67b894fec79d [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: NON_VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: true } }
I20240501 14:09:29.521076 11574 raft_consensus.cc:729] T c747bd6504c540cebded0019d82495b4 P b4105a6d531d401faadf67b894fec79d [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: b4105a6d531d401faadf67b894fec79d, State: Initialized, Role: LEARNER
I20240501 14:09:29.521157 11574 consensus_queue.cc:260] T c747bd6504c540cebded0019d82495b4 P b4105a6d531d401faadf67b894fec79d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: NON_VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: true } }
I20240501 14:09:29.521478 11574 ts_tablet_manager.cc:1398] T c747bd6504c540cebded0019d82495b4 P b4105a6d531d401faadf67b894fec79d: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 14:09:29.522047 7793 tablet_copy_service.cc:342] P 9ae5a87ca07b40a788a42734b24f66c6: Request end of tablet copy session b4105a6d531d401faadf67b894fec79d-c747bd6504c540cebded0019d82495b4 received from {username='slave'} at 127.7.39.135:36239
I20240501 14:09:29.522145 7793 tablet_copy_service.cc:433] P 9ae5a87ca07b40a788a42734b24f66c6: ending tablet copy session b4105a6d531d401faadf67b894fec79d-c747bd6504c540cebded0019d82495b4 on tablet c747bd6504c540cebded0019d82495b4 with peer b4105a6d531d401faadf67b894fec79d
I20240501 14:09:29.569080 11659 consensus_queue.cc:579] T 2880045de45743bdb5358a17d93b2245 P 9ae5a87ca07b40a788a42734b24f66c6 [LEADER]: Leader has been unable to successfully communicate with peer ae52f888f5094416a24cb3dffbff03c4 for more than 30 seconds (30.185s)
I20240501 14:09:29.583860 11648 consensus_queue.cc:579] T 14e325d8d3184c9192b8cebafb414190 P 47eabc135e9f4237b036097085e729d8 [LEADER]: Leader has been unable to successfully communicate with peer ae52f888f5094416a24cb3dffbff03c4 for more than 30 seconds (30.215s)
I20240501 14:09:29.796272 9215 raft_consensus.cc:1203] T c747bd6504c540cebded0019d82495b4 P b4105a6d531d401faadf67b894fec79d [term 2 LEARNER]: Deduplicated request from leader. Original: 2.11->[2.12-2.12] Dedup: 2.12->[]
I20240501 14:09:29.832568 8953 raft_consensus.cc:1203] T 6841e1b32b86454f9e40b64c141a2a6b P 9faf312806f54f2e8a4a807b234e0d1b [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
W20240501 14:09:30.017428 7839 consensus_peers.cc:485] T 14e325d8d3184c9192b8cebafb414190 P 47eabc135e9f4237b036097085e729d8 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 14:09:30.077111 11659 consensus_queue.cc:579] T dcf673c75a8b48da95806e1c43309dda P 9ae5a87ca07b40a788a42734b24f66c6 [LEADER]: Leader has been unable to successfully communicate with peer ae52f888f5094416a24cb3dffbff03c4 for more than 30 seconds (30.529s)
W20240501 14:09:30.118588 7839 consensus_peers.cc:485] T 4d282d79e1664f7ea8912e0c5166eec3 P 47eabc135e9f4237b036097085e729d8 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 14:09:30.195096 11659 raft_consensus.cc:1050] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6: attempting to promote NON_VOTER b4105a6d531d401faadf67b894fec79d to VOTER
I20240501 14:09:30.195545 11659 consensus_queue.cc:237] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } }
I20240501 14:09:30.196998 9346 raft_consensus.cc:1261] T c747bd6504c540cebded0019d82495b4 P 9d32e843754e44d0b9aae6f472a4d9a2 [term 2 FOLLOWER]: Refusing update from remote peer 9ae5a87ca07b40a788a42734b24f66c6: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 14:09:30.196998 9083 raft_consensus.cc:1261] T c747bd6504c540cebded0019d82495b4 P 9d7c8c0f7907428eab22633702ead7ab [term 2 FOLLOWER]: Refusing update from remote peer 9ae5a87ca07b40a788a42734b24f66c6: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 14:09:30.197432 11696 consensus_queue.cc:1035] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:09:30.197808 9215 raft_consensus.cc:1261] T c747bd6504c540cebded0019d82495b4 P b4105a6d531d401faadf67b894fec79d [term 2 LEARNER]: Refusing update from remote peer 9ae5a87ca07b40a788a42734b24f66c6: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 14:09:30.197798 11696 consensus_queue.cc:1035] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:09:30.198179 11709 consensus_queue.cc:1035] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:09:30.200467 11709 raft_consensus.cc:2932] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, b4105a6d531d401faadf67b894fec79d (127.7.39.135) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } }
I20240501 14:09:30.201090 9083 raft_consensus.cc:2932] T c747bd6504c540cebded0019d82495b4 P 9d7c8c0f7907428eab22633702ead7ab [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, b4105a6d531d401faadf67b894fec79d (127.7.39.135) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } }
I20240501 14:09:30.202826 9346 raft_consensus.cc:2932] T c747bd6504c540cebded0019d82495b4 P 9d32e843754e44d0b9aae6f472a4d9a2 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, b4105a6d531d401faadf67b894fec79d (127.7.39.135) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } }
I20240501 14:09:30.203567 9215 raft_consensus.cc:2932] T c747bd6504c540cebded0019d82495b4 P b4105a6d531d401faadf67b894fec79d [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, b4105a6d531d401faadf67b894fec79d (127.7.39.135) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } }
I20240501 14:09:30.205192 7366 catalog_manager.cc:5484] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 reported cstate change: config changed from index 12 to 13, b4105a6d531d401faadf67b894fec79d (127.7.39.135) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "9ae5a87ca07b40a788a42734b24f66c6" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:30.300859 11719 raft_consensus.cc:1050] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634: attempting to promote NON_VOTER 9faf312806f54f2e8a4a807b234e0d1b to VOTER
I20240501 14:09:30.301363 11719 consensus_queue.cc:237] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } }
I20240501 14:09:30.302698 8953 raft_consensus.cc:1261] T 6841e1b32b86454f9e40b64c141a2a6b P 9faf312806f54f2e8a4a807b234e0d1b [term 2 LEARNER]: Refusing update from remote peer 9e44efa72e7d4421b284ae74d1cb9634: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:09:30.302824 9083 raft_consensus.cc:1261] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab [term 2 FOLLOWER]: Refusing update from remote peer 9e44efa72e7d4421b284ae74d1cb9634: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:09:30.302965 7904 raft_consensus.cc:1261] T 6841e1b32b86454f9e40b64c141a2a6b P 47eabc135e9f4237b036097085e729d8 [term 2 FOLLOWER]: Refusing update from remote peer 9e44efa72e7d4421b284ae74d1cb9634: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:09:30.303086 11721 consensus_queue.cc:1035] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:30.303241 11603 consensus_queue.cc:1035] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:30.303364 11719 consensus_queue.cc:1035] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:30.305186 11637 raft_consensus.cc:2932] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 9faf312806f54f2e8a4a807b234e0d1b (127.7.39.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } }
I20240501 14:09:30.305564 8953 raft_consensus.cc:2932] T 6841e1b32b86454f9e40b64c141a2a6b P 9faf312806f54f2e8a4a807b234e0d1b [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 9faf312806f54f2e8a4a807b234e0d1b (127.7.39.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } }
I20240501 14:09:30.305611 7904 raft_consensus.cc:2932] T 6841e1b32b86454f9e40b64c141a2a6b P 47eabc135e9f4237b036097085e729d8 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 9faf312806f54f2e8a4a807b234e0d1b (127.7.39.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } }
I20240501 14:09:30.308192 7369 catalog_manager.cc:5484] T 6841e1b32b86454f9e40b64c141a2a6b P 9faf312806f54f2e8a4a807b234e0d1b reported cstate change: config changed from index 8 to 9, 9faf312806f54f2e8a4a807b234e0d1b (127.7.39.133) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "9e44efa72e7d4421b284ae74d1cb9634" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } }
I20240501 14:09:30.308136 9083 raft_consensus.cc:2932] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 9faf312806f54f2e8a4a807b234e0d1b (127.7.39.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } }
I20240501 14:09:30.401736 7771 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c747bd6504c540cebded0019d82495b4"
dest_uuid: "9ae5a87ca07b40a788a42734b24f66c6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35620
I20240501 14:09:30.402105 7771 raft_consensus.cc:595] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [term 2 LEADER]: Received request to transfer leadership
I20240501 14:09:30.414302 7642 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6841e1b32b86454f9e40b64c141a2a6b"
dest_uuid: "9e44efa72e7d4421b284ae74d1cb9634"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47196
I20240501 14:09:30.414467 7642 raft_consensus.cc:595] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634 [term 2 LEADER]: Received request to transfer leadership
W20240501 14:09:30.432888 7708 consensus_peers.cc:485] T ceee52d679bf4d8e8896a6c0dde0519b P 9ae5a87ca07b40a788a42734b24f66c6 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:09:30.489041 7708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 14:09:30.625891 7771 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c747bd6504c540cebded0019d82495b4"
dest_uuid: "9ae5a87ca07b40a788a42734b24f66c6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35638
I20240501 14:09:30.626049 7771 raft_consensus.cc:595] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [term 2 LEADER]: Received request to transfer leadership
W20240501 14:09:30.628854 7839 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240501 14:09:30.630841 7577 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240501 14:09:30.636850 7642 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6841e1b32b86454f9e40b64c141a2a6b"
dest_uuid: "9e44efa72e7d4421b284ae74d1cb9634"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47210
I20240501 14:09:30.636979 7642 raft_consensus.cc:595] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634 [term 2 LEADER]: Received request to transfer leadership
I20240501 14:09:30.723779 11659 raft_consensus.cc:979] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6: : Instructing follower 9d32e843754e44d0b9aae6f472a4d9a2 to start an election
I20240501 14:09:30.724087 11659 raft_consensus.cc:1067] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [term 2 LEADER]: Signalling peer 9d32e843754e44d0b9aae6f472a4d9a2 to start an election
I20240501 14:09:30.724602 9346 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c747bd6504c540cebded0019d82495b4"
dest_uuid: "9d32e843754e44d0b9aae6f472a4d9a2"
from {username='slave'} at 127.7.39.131:35277
I20240501 14:09:30.724807 9346 raft_consensus.cc:482] T c747bd6504c540cebded0019d82495b4 P 9d32e843754e44d0b9aae6f472a4d9a2 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:30.724884 9346 raft_consensus.cc:3037] T c747bd6504c540cebded0019d82495b4 P 9d32e843754e44d0b9aae6f472a4d9a2 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:09:30.725733 9346 raft_consensus.cc:504] T c747bd6504c540cebded0019d82495b4 P 9d32e843754e44d0b9aae6f472a4d9a2 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } }
I20240501 14:09:30.726791 9346 leader_election.cc:288] T c747bd6504c540cebded0019d82495b4 P 9d32e843754e44d0b9aae6f472a4d9a2 [CANDIDATE]: Term 3 election: Requested vote from peers 9ae5a87ca07b40a788a42734b24f66c6 (127.7.39.131:34697), 9d7c8c0f7907428eab22633702ead7ab (127.7.39.134:38699), b4105a6d531d401faadf67b894fec79d (127.7.39.135:39611)
I20240501 14:09:30.731132 9215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c747bd6504c540cebded0019d82495b4" candidate_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: true dest_uuid: "b4105a6d531d401faadf67b894fec79d"
I20240501 14:09:30.731267 9215 raft_consensus.cc:3037] T c747bd6504c540cebded0019d82495b4 P b4105a6d531d401faadf67b894fec79d [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:09:30.731392 7771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c747bd6504c540cebded0019d82495b4" candidate_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: true dest_uuid: "9ae5a87ca07b40a788a42734b24f66c6"
I20240501 14:09:30.731511 7771 raft_consensus.cc:3032] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:09:30.731550 7771 raft_consensus.cc:729] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9ae5a87ca07b40a788a42734b24f66c6, State: Running, Role: LEADER
I20240501 14:09:30.731707 7771 consensus_queue.cc:260] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } }
I20240501 14:09:30.731902 7771 raft_consensus.cc:3037] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:09:30.732245 9083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c747bd6504c540cebded0019d82495b4" candidate_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: true dest_uuid: "9d7c8c0f7907428eab22633702ead7ab"
I20240501 14:09:30.732345 9083 raft_consensus.cc:3037] T c747bd6504c540cebded0019d82495b4 P 9d7c8c0f7907428eab22633702ead7ab [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:09:30.732399 9215 raft_consensus.cc:2454] T c747bd6504c540cebded0019d82495b4 P b4105a6d531d401faadf67b894fec79d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d32e843754e44d0b9aae6f472a4d9a2 in term 3.
I20240501 14:09:30.732945 7771 raft_consensus.cc:2454] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d32e843754e44d0b9aae6f472a4d9a2 in term 3.
I20240501 14:09:30.733222 9083 raft_consensus.cc:2454] T c747bd6504c540cebded0019d82495b4 P 9d7c8c0f7907428eab22633702ead7ab [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d32e843754e44d0b9aae6f472a4d9a2 in term 3.
I20240501 14:09:30.733282 9281 leader_election.cc:302] T c747bd6504c540cebded0019d82495b4 P 9d32e843754e44d0b9aae6f472a4d9a2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 9ae5a87ca07b40a788a42734b24f66c6, 9d32e843754e44d0b9aae6f472a4d9a2, b4105a6d531d401faadf67b894fec79d; no voters:
I20240501 14:09:30.733685 11773 raft_consensus.cc:2781] T c747bd6504c540cebded0019d82495b4 P 9d32e843754e44d0b9aae6f472a4d9a2 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:09:30.733919 11773 raft_consensus.cc:686] T c747bd6504c540cebded0019d82495b4 P 9d32e843754e44d0b9aae6f472a4d9a2 [term 3 LEADER]: Becoming Leader. State: Replica: 9d32e843754e44d0b9aae6f472a4d9a2, State: Running, Role: LEADER
I20240501 14:09:30.734288 11773 consensus_queue.cc:237] T c747bd6504c540cebded0019d82495b4 P 9d32e843754e44d0b9aae6f472a4d9a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } }
I20240501 14:09:30.737120 7367 catalog_manager.cc:5484] T c747bd6504c540cebded0019d82495b4 P 9d32e843754e44d0b9aae6f472a4d9a2 reported cstate change: term changed from 2 to 3, leader changed from 9ae5a87ca07b40a788a42734b24f66c6 (127.7.39.131) to 9d32e843754e44d0b9aae6f472a4d9a2 (127.7.39.136). New cstate: current_term: 3 leader_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:30.809674 11719 raft_consensus.cc:979] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634: : Instructing follower 9d7c8c0f7907428eab22633702ead7ab to start an election
I20240501 14:09:30.809830 11719 raft_consensus.cc:1067] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634 [term 2 LEADER]: Signalling peer 9d7c8c0f7907428eab22633702ead7ab to start an election
I20240501 14:09:30.810407 9083 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6841e1b32b86454f9e40b64c141a2a6b"
dest_uuid: "9d7c8c0f7907428eab22633702ead7ab"
from {username='slave'} at 127.7.39.130:51049
I20240501 14:09:30.810609 9083 raft_consensus.cc:482] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:30.810679 9083 raft_consensus.cc:3037] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:09:30.811579 9083 raft_consensus.cc:504] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } }
I20240501 14:09:30.812610 9083 leader_election.cc:288] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab [CANDIDATE]: Term 3 election: Requested vote from peers 9e44efa72e7d4421b284ae74d1cb9634 (127.7.39.130:46433), 47eabc135e9f4237b036097085e729d8 (127.7.39.132:42369), 9faf312806f54f2e8a4a807b234e0d1b (127.7.39.133:38805)
I20240501 14:09:30.812912 7642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6841e1b32b86454f9e40b64c141a2a6b" candidate_uuid: "9d7c8c0f7907428eab22633702ead7ab" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: true dest_uuid: "9e44efa72e7d4421b284ae74d1cb9634"
I20240501 14:09:30.813042 7642 raft_consensus.cc:3032] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:09:30.813094 7642 raft_consensus.cc:729] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9e44efa72e7d4421b284ae74d1cb9634, State: Running, Role: LEADER
I20240501 14:09:30.813035 7904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6841e1b32b86454f9e40b64c141a2a6b" candidate_uuid: "9d7c8c0f7907428eab22633702ead7ab" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: true dest_uuid: "47eabc135e9f4237b036097085e729d8"
I20240501 14:09:30.813180 7904 raft_consensus.cc:3037] T 6841e1b32b86454f9e40b64c141a2a6b P 47eabc135e9f4237b036097085e729d8 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:09:30.813264 7642 consensus_queue.cc:260] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } }
I20240501 14:09:30.813475 7642 raft_consensus.cc:3037] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:09:30.814265 7904 raft_consensus.cc:2454] T 6841e1b32b86454f9e40b64c141a2a6b P 47eabc135e9f4237b036097085e729d8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d7c8c0f7907428eab22633702ead7ab in term 3.
I20240501 14:09:30.814486 7642 raft_consensus.cc:2454] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d7c8c0f7907428eab22633702ead7ab in term 3.
I20240501 14:09:30.814818 9019 leader_election.cc:302] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 47eabc135e9f4237b036097085e729d8, 9d7c8c0f7907428eab22633702ead7ab, 9e44efa72e7d4421b284ae74d1cb9634; no voters:
I20240501 14:09:30.815164 11777 raft_consensus.cc:2781] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:09:30.815336 11777 raft_consensus.cc:686] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab [term 3 LEADER]: Becoming Leader. State: Replica: 9d7c8c0f7907428eab22633702ead7ab, State: Running, Role: LEADER
I20240501 14:09:30.815637 11777 consensus_queue.cc:237] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } }
I20240501 14:09:30.817771 8953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6841e1b32b86454f9e40b64c141a2a6b" candidate_uuid: "9d7c8c0f7907428eab22633702ead7ab" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: true dest_uuid: "9faf312806f54f2e8a4a807b234e0d1b"
I20240501 14:09:30.817914 8953 raft_consensus.cc:3037] T 6841e1b32b86454f9e40b64c141a2a6b P 9faf312806f54f2e8a4a807b234e0d1b [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:09:30.818202 7367 catalog_manager.cc:5484] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab reported cstate change: term changed from 2 to 3, leader changed from 9e44efa72e7d4421b284ae74d1cb9634 (127.7.39.130) to 9d7c8c0f7907428eab22633702ead7ab (127.7.39.134). New cstate: current_term: 3 leader_uuid: "9d7c8c0f7907428eab22633702ead7ab" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:30.818912 8953 raft_consensus.cc:2454] T 6841e1b32b86454f9e40b64c141a2a6b P 9faf312806f54f2e8a4a807b234e0d1b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d7c8c0f7907428eab22633702ead7ab in term 3.
W20240501 14:09:30.979364 7708 consensus_peers.cc:485] T 2880045de45743bdb5358a17d93b2245 P 9ae5a87ca07b40a788a42734b24f66c6 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 14:09:31.002158 11659 consensus_queue.cc:579] T ceee52d679bf4d8e8896a6c0dde0519b P 9ae5a87ca07b40a788a42734b24f66c6 [LEADER]: Leader has been unable to successfully communicate with peer ae52f888f5094416a24cb3dffbff03c4 for more than 30 seconds (30.039s)
W20240501 14:09:31.090720 7355 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 14:09:31.091564 7355 catalog_manager.cc:4573] TS ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 2112bf96dc3343678aa6f46bf2a3b1f2: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111)
I20240501 14:09:31.098919 11719 consensus_queue.cc:579] T df856440b07b418394477cf368b765c1 P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Leader has been unable to successfully communicate with peer ae52f888f5094416a24cb3dffbff03c4 for more than 30 seconds (30.036s)
I20240501 14:09:31.117480 9083 raft_consensus.cc:1261] T c747bd6504c540cebded0019d82495b4 P 9d7c8c0f7907428eab22633702ead7ab [term 3 FOLLOWER]: Refusing update from remote peer 9d32e843754e44d0b9aae6f472a4d9a2: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240501 14:09:31.118042 11773 consensus_queue.cc:1035] T c747bd6504c540cebded0019d82495b4 P 9d32e843754e44d0b9aae6f472a4d9a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 14:09:31.124704 7771 raft_consensus.cc:1261] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [term 3 FOLLOWER]: Refusing update from remote peer 9d32e843754e44d0b9aae6f472a4d9a2: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240501 14:09:31.125209 11773 consensus_queue.cc:1035] T c747bd6504c540cebded0019d82495b4 P 9d32e843754e44d0b9aae6f472a4d9a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 14:09:31.128477 9215 raft_consensus.cc:1261] T c747bd6504c540cebded0019d82495b4 P b4105a6d531d401faadf67b894fec79d [term 3 FOLLOWER]: Refusing update from remote peer 9d32e843754e44d0b9aae6f472a4d9a2: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240501 14:09:31.128974 11805 consensus_queue.cc:1035] T c747bd6504c540cebded0019d82495b4 P 9d32e843754e44d0b9aae6f472a4d9a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 14:09:31.151023 11803 consensus_queue.cc:579] T 2cf2f85819584fdf872f6c590d11f873 P 9e44efa72e7d4421b284ae74d1cb9634 [LEADER]: Leader has been unable to successfully communicate with peer ae52f888f5094416a24cb3dffbff03c4 for more than 30 seconds (30.380s)
I20240501 14:09:31.343969 7904 raft_consensus.cc:1261] T 6841e1b32b86454f9e40b64c141a2a6b P 47eabc135e9f4237b036097085e729d8 [term 3 FOLLOWER]: Refusing update from remote peer 9d7c8c0f7907428eab22633702ead7ab: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240501 14:09:31.344482 11777 consensus_queue.cc:1035] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:09:31.369927 8953 raft_consensus.cc:1261] T 6841e1b32b86454f9e40b64c141a2a6b P 9faf312806f54f2e8a4a807b234e0d1b [term 3 FOLLOWER]: Refusing update from remote peer 9d7c8c0f7907428eab22633702ead7ab: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240501 14:09:31.370342 11777 consensus_queue.cc:1035] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:09:31.373463 7642 raft_consensus.cc:1261] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634 [term 3 FOLLOWER]: Refusing update from remote peer 9d7c8c0f7907428eab22633702ead7ab: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240501 14:09:31.375391 11823 consensus_queue.cc:1035] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:09:31.379113 9083 consensus_queue.cc:237] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 3.10, Last appended by leader: 9, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } }
I20240501 14:09:31.380129 8953 raft_consensus.cc:1261] T 6841e1b32b86454f9e40b64c141a2a6b P 9faf312806f54f2e8a4a807b234e0d1b [term 3 FOLLOWER]: Refusing update from remote peer 9d7c8c0f7907428eab22633702ead7ab: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240501 14:09:31.380192 7904 raft_consensus.cc:1261] T 6841e1b32b86454f9e40b64c141a2a6b P 47eabc135e9f4237b036097085e729d8 [term 3 FOLLOWER]: Refusing update from remote peer 9d7c8c0f7907428eab22633702ead7ab: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240501 14:09:31.380504 11826 consensus_queue.cc:1035] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:09:31.380641 11823 consensus_queue.cc:1035] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:09:31.381981 11824 raft_consensus.cc:2932] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab [term 3 LEADER]: Committing config change with OpId 3.11: config changed from index 9 to 11, VOTER 9e44efa72e7d4421b284ae74d1cb9634 (127.7.39.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } }
I20240501 14:09:31.382475 7904 raft_consensus.cc:2932] T 6841e1b32b86454f9e40b64c141a2a6b P 47eabc135e9f4237b036097085e729d8 [term 3 FOLLOWER]: Committing config change with OpId 3.11: config changed from index 9 to 11, VOTER 9e44efa72e7d4421b284ae74d1cb9634 (127.7.39.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } }
I20240501 14:09:31.382503 8953 raft_consensus.cc:2932] T 6841e1b32b86454f9e40b64c141a2a6b P 9faf312806f54f2e8a4a807b234e0d1b [term 3 FOLLOWER]: Committing config change with OpId 3.11: config changed from index 9 to 11, VOTER 9e44efa72e7d4421b284ae74d1cb9634 (127.7.39.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } }
I20240501 14:09:31.383898 7356 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6841e1b32b86454f9e40b64c141a2a6b with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:31.384809 7367 catalog_manager.cc:5484] T 6841e1b32b86454f9e40b64c141a2a6b P 47eabc135e9f4237b036097085e729d8 reported cstate change: config changed from index 9 to 11, VOTER 9e44efa72e7d4421b284ae74d1cb9634 (127.7.39.130) evicted. New cstate: current_term: 3 leader_uuid: "9d7c8c0f7907428eab22633702ead7ab" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } }
I20240501 14:09:31.387439 7620 tablet_service.cc:1514] Processing DeleteTablet for tablet 6841e1b32b86454f9e40b64c141a2a6b with delete_type TABLET_DATA_TOMBSTONED (TS 9e44efa72e7d4421b284ae74d1cb9634 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:60602
I20240501 14:09:31.387814 11833 tablet_replica.cc:330] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634: stopping tablet replica
I20240501 14:09:31.387948 11833 raft_consensus.cc:2229] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:31.388062 11833 raft_consensus.cc:2258] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:31.388674 11833 ts_tablet_manager.cc:1875] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:31.390458 11833 ts_tablet_manager.cc:1888] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.10
I20240501 14:09:31.390532 11833 log.cc:1196] T 6841e1b32b86454f9e40b64c141a2a6b P 9e44efa72e7d4421b284ae74d1cb9634: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal/wals/6841e1b32b86454f9e40b64c141a2a6b
I20240501 14:09:31.391134 7355 catalog_manager.cc:4830] TS 9e44efa72e7d4421b284ae74d1cb9634 (127.7.39.130:46433): tablet 6841e1b32b86454f9e40b64c141a2a6b (table rebalance_test_table_3 [id=1efd7e27dfbf4a88a4c2c2d4d7ebfe4f]) successfully deleted
W20240501 14:09:31.523993 7577 consensus_peers.cc:485] T df856440b07b418394477cf368b765c1 P 9e44efa72e7d4421b284ae74d1cb9634 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:09:31.621196 7577 consensus_peers.cc:485] T 2cf2f85819584fdf872f6c590d11f873 P 9e44efa72e7d4421b284ae74d1cb9634 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 14:09:31.688071 9346 consensus_queue.cc:237] T c747bd6504c540cebded0019d82495b4 P 9d32e843754e44d0b9aae6f472a4d9a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 3.14, Last appended by leader: 13, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } }
I20240501 14:09:31.689378 9083 raft_consensus.cc:1261] T c747bd6504c540cebded0019d82495b4 P 9d7c8c0f7907428eab22633702ead7ab [term 3 FOLLOWER]: Refusing update from remote peer 9d32e843754e44d0b9aae6f472a4d9a2: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 3 index: 15. (index mismatch)
I20240501 14:09:31.689343 9215 raft_consensus.cc:1261] T c747bd6504c540cebded0019d82495b4 P b4105a6d531d401faadf67b894fec79d [term 3 FOLLOWER]: Refusing update from remote peer 9d32e843754e44d0b9aae6f472a4d9a2: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 3 index: 15. (index mismatch)
I20240501 14:09:31.689827 11849 consensus_queue.cc:1035] T c747bd6504c540cebded0019d82495b4 P 9d32e843754e44d0b9aae6f472a4d9a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240501 14:09:31.690052 11850 consensus_queue.cc:1035] T c747bd6504c540cebded0019d82495b4 P 9d32e843754e44d0b9aae6f472a4d9a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240501 14:09:31.691630 11774 raft_consensus.cc:2932] T c747bd6504c540cebded0019d82495b4 P 9d32e843754e44d0b9aae6f472a4d9a2 [term 3 LEADER]: Committing config change with OpId 3.15: config changed from index 13 to 15, VOTER 9ae5a87ca07b40a788a42734b24f66c6 (127.7.39.131) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } }
I20240501 14:09:31.692003 9083 raft_consensus.cc:2932] T c747bd6504c540cebded0019d82495b4 P 9d7c8c0f7907428eab22633702ead7ab [term 3 FOLLOWER]: Committing config change with OpId 3.15: config changed from index 13 to 15, VOTER 9ae5a87ca07b40a788a42734b24f66c6 (127.7.39.131) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } }
I20240501 14:09:31.693080 7354 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c747bd6504c540cebded0019d82495b4 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:31.693761 9215 raft_consensus.cc:2932] T c747bd6504c540cebded0019d82495b4 P b4105a6d531d401faadf67b894fec79d [term 3 FOLLOWER]: Committing config change with OpId 3.15: config changed from index 13 to 15, VOTER 9ae5a87ca07b40a788a42734b24f66c6 (127.7.39.131) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } }
I20240501 14:09:31.694217 7369 catalog_manager.cc:5484] T c747bd6504c540cebded0019d82495b4 P 9d7c8c0f7907428eab22633702ead7ab reported cstate change: config changed from index 13 to 15, VOTER 9ae5a87ca07b40a788a42734b24f66c6 (127.7.39.131) evicted. New cstate: current_term: 3 leader_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "b4105a6d531d401faadf67b894fec79d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 39611 } attrs { promote: false } } }
I20240501 14:09:31.697028 7752 tablet_service.cc:1514] Processing DeleteTablet for tablet c747bd6504c540cebded0019d82495b4 with delete_type TABLET_DATA_TOMBSTONED (TS 9ae5a87ca07b40a788a42734b24f66c6 not found in new config with opid_index 15) from {username='slave'} at 127.0.0.1:36366
I20240501 14:09:31.697837 11854 tablet_replica.cc:330] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6: stopping tablet replica
I20240501 14:09:31.698029 11854 raft_consensus.cc:2229] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:31.698150 11854 raft_consensus.cc:2258] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:31.698798 11854 ts_tablet_manager.cc:1875] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:31.701129 11854 ts_tablet_manager.cc:1888] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.14
I20240501 14:09:31.701222 11854 log.cc:1196] T c747bd6504c540cebded0019d82495b4 P 9ae5a87ca07b40a788a42734b24f66c6: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal/wals/c747bd6504c540cebded0019d82495b4
I20240501 14:09:31.701799 7355 catalog_manager.cc:4830] TS 9ae5a87ca07b40a788a42734b24f66c6 (127.7.39.131:34697): tablet c747bd6504c540cebded0019d82495b4 (table rebalance_test_table_4 [id=9b4ecb4938fd4bc185e20ce3ec5254f5]) successfully deleted
I20240501 14:09:31.771261 9326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:31.772418 8931 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:31.775998 7752 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:31.778695 7620 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:31.782967 9064 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:31.787613 7882 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:31.788763 9195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:31.830260 9083 consensus_queue.cc:237] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 3.11, Last appended by leader: 9, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: NON_VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: true } }
I20240501 14:09:31.832108 7904 raft_consensus.cc:1261] T 6841e1b32b86454f9e40b64c141a2a6b P 47eabc135e9f4237b036097085e729d8 [term 3 FOLLOWER]: Refusing update from remote peer 9d7c8c0f7907428eab22633702ead7ab: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240501 14:09:31.832157 8953 raft_consensus.cc:1261] T 6841e1b32b86454f9e40b64c141a2a6b P 9faf312806f54f2e8a4a807b234e0d1b [term 3 FOLLOWER]: Refusing update from remote peer 9d7c8c0f7907428eab22633702ead7ab: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240501 14:09:31.832540 11823 consensus_queue.cc:1035] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:09:31.832711 11777 consensus_queue.cc:1035] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:09:31.834623 11823 raft_consensus.cc:2932] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab [term 3 LEADER]: Committing config change with OpId 3.12: config changed from index 11 to 12, NON_VOTER 9d32e843754e44d0b9aae6f472a4d9a2 (127.7.39.136) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: NON_VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: true } } }
I20240501 14:09:31.835420 8953 raft_consensus.cc:2932] T 6841e1b32b86454f9e40b64c141a2a6b P 9faf312806f54f2e8a4a807b234e0d1b [term 3 FOLLOWER]: Committing config change with OpId 3.12: config changed from index 11 to 12, NON_VOTER 9d32e843754e44d0b9aae6f472a4d9a2 (127.7.39.136) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: NON_VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: true } } }
W20240501 14:09:31.835705 9018 consensus_peers.cc:485] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab -> Peer 9d32e843754e44d0b9aae6f472a4d9a2 (127.7.39.136:37475): Couldn't send request to peer 9d32e843754e44d0b9aae6f472a4d9a2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6841e1b32b86454f9e40b64c141a2a6b. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:31.836784 7904 raft_consensus.cc:2932] T 6841e1b32b86454f9e40b64c141a2a6b P 47eabc135e9f4237b036097085e729d8 [term 3 FOLLOWER]: Committing config change with OpId 3.12: config changed from index 11 to 12, NON_VOTER 9d32e843754e44d0b9aae6f472a4d9a2 (127.7.39.136) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: NON_VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: true } } }
I20240501 14:09:31.838070 7366 catalog_manager.cc:5484] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab reported cstate change: config changed from index 11 to 12, NON_VOTER 9d32e843754e44d0b9aae6f472a4d9a2 (127.7.39.136) added. New cstate: current_term: 3 leader_uuid: "9d7c8c0f7907428eab22633702ead7ab" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: NON_VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:32.240015 7708 consensus_peers.cc:485] T dcf673c75a8b48da95806e1c43309dda P 9ae5a87ca07b40a788a42734b24f66c6 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 14:09:32.242406 11915 ts_tablet_manager.cc:902] T 6841e1b32b86454f9e40b64c141a2a6b P 9d32e843754e44d0b9aae6f472a4d9a2: Initiating tablet copy from peer 9d7c8c0f7907428eab22633702ead7ab (127.7.39.134:38699)
I20240501 14:09:32.242934 11915 tablet_copy_client.cc:308] T 6841e1b32b86454f9e40b64c141a2a6b P 9d32e843754e44d0b9aae6f472a4d9a2: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.134:38699
I20240501 14:09:32.243579 9104 tablet_copy_service.cc:140] P 9d7c8c0f7907428eab22633702ead7ab: Received BeginTabletCopySession request for tablet 6841e1b32b86454f9e40b64c141a2a6b from peer 9d32e843754e44d0b9aae6f472a4d9a2 ({username='slave'} at 127.7.39.136:47161)
I20240501 14:09:32.243728 9104 tablet_copy_service.cc:161] P 9d7c8c0f7907428eab22633702ead7ab: Beginning new tablet copy session on tablet 6841e1b32b86454f9e40b64c141a2a6b from peer 9d32e843754e44d0b9aae6f472a4d9a2 at {username='slave'} at 127.7.39.136:47161: session id = 9d32e843754e44d0b9aae6f472a4d9a2-6841e1b32b86454f9e40b64c141a2a6b
I20240501 14:09:32.244792 9104 tablet_copy_source_session.cc:205] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:32.245827 11915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6841e1b32b86454f9e40b64c141a2a6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:32.247635 11915 tablet_copy_client.cc:783] T 6841e1b32b86454f9e40b64c141a2a6b P 9d32e843754e44d0b9aae6f472a4d9a2: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:32.247773 11915 tablet_copy_client.cc:647] T 6841e1b32b86454f9e40b64c141a2a6b P 9d32e843754e44d0b9aae6f472a4d9a2: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:32.249125 11917 tablet_copy_client.cc:1002] T 6841e1b32b86454f9e40b64c141a2a6b P 9d32e843754e44d0b9aae6f472a4d9a2: tablet copy: Injecting latency into file download: 1500
W20240501 14:09:32.754947 7839 consensus_peers.cc:485] T 14e325d8d3184c9192b8cebafb414190 P 47eabc135e9f4237b036097085e729d8 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:09:32.825199 7708 consensus_peers.cc:485] T ceee52d679bf4d8e8896a6c0dde0519b P 9ae5a87ca07b40a788a42734b24f66c6 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:09:32.899271 7839 consensus_peers.cc:485] T 4d282d79e1664f7ea8912e0c5166eec3 P 47eabc135e9f4237b036097085e729d8 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:09:33.327515 7708 consensus_peers.cc:485] T 2880045de45743bdb5358a17d93b2245 P 9ae5a87ca07b40a788a42734b24f66c6 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 14:09:33.749477 11915 tablet_copy_client.cc:523] T 6841e1b32b86454f9e40b64c141a2a6b P 9d32e843754e44d0b9aae6f472a4d9a2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:33.751102 11915 tablet_bootstrap.cc:492] T 6841e1b32b86454f9e40b64c141a2a6b P 9d32e843754e44d0b9aae6f472a4d9a2: Bootstrap starting.
I20240501 14:09:33.770354 11915 tablet_bootstrap.cc:492] T 6841e1b32b86454f9e40b64c141a2a6b P 9d32e843754e44d0b9aae6f472a4d9a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:33.770651 11915 tablet_bootstrap.cc:492] T 6841e1b32b86454f9e40b64c141a2a6b P 9d32e843754e44d0b9aae6f472a4d9a2: Bootstrap complete.
I20240501 14:09:33.770841 11915 ts_tablet_manager.cc:1367] T 6841e1b32b86454f9e40b64c141a2a6b P 9d32e843754e44d0b9aae6f472a4d9a2: Time spent bootstrapping tablet: real 0.020s user 0.014s sys 0.003s
I20240501 14:09:33.771322 11915 raft_consensus.cc:348] T 6841e1b32b86454f9e40b64c141a2a6b P 9d32e843754e44d0b9aae6f472a4d9a2 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: NON_VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: true } }
I20240501 14:09:33.771437 11915 raft_consensus.cc:729] T 6841e1b32b86454f9e40b64c141a2a6b P 9d32e843754e44d0b9aae6f472a4d9a2 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 9d32e843754e44d0b9aae6f472a4d9a2, State: Initialized, Role: LEARNER
I20240501 14:09:33.771528 11915 consensus_queue.cc:260] T 6841e1b32b86454f9e40b64c141a2a6b P 9d32e843754e44d0b9aae6f472a4d9a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: NON_VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: true } }
I20240501 14:09:33.771818 11915 ts_tablet_manager.cc:1398] T 6841e1b32b86454f9e40b64c141a2a6b P 9d32e843754e44d0b9aae6f472a4d9a2: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 14:09:33.772500 9104 tablet_copy_service.cc:342] P 9d7c8c0f7907428eab22633702ead7ab: Request end of tablet copy session 9d32e843754e44d0b9aae6f472a4d9a2-6841e1b32b86454f9e40b64c141a2a6b received from {username='slave'} at 127.7.39.136:47161
I20240501 14:09:33.772624 9104 tablet_copy_service.cc:433] P 9d7c8c0f7907428eab22633702ead7ab: ending tablet copy session 9d32e843754e44d0b9aae6f472a4d9a2-6841e1b32b86454f9e40b64c141a2a6b on tablet 6841e1b32b86454f9e40b64c141a2a6b with peer 9d32e843754e44d0b9aae6f472a4d9a2
I20240501 14:09:34.152601 9346 raft_consensus.cc:1203] T 6841e1b32b86454f9e40b64c141a2a6b P 9d32e843754e44d0b9aae6f472a4d9a2 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.11->[3.12-3.12] Dedup: 3.12->[]
W20240501 14:09:34.190853 7577 consensus_peers.cc:485] T df856440b07b418394477cf368b765c1 P 9e44efa72e7d4421b284ae74d1cb9634 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:09:34.224701 7577 consensus_peers.cc:485] T 2cf2f85819584fdf872f6c590d11f873 P 9e44efa72e7d4421b284ae74d1cb9634 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 14:09:34.626581 11986 raft_consensus.cc:1050] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab: attempting to promote NON_VOTER 9d32e843754e44d0b9aae6f472a4d9a2 to VOTER
I20240501 14:09:34.627094 11986 consensus_queue.cc:237] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 9, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } }
I20240501 14:09:34.628479 9346 raft_consensus.cc:1261] T 6841e1b32b86454f9e40b64c141a2a6b P 9d32e843754e44d0b9aae6f472a4d9a2 [term 3 LEARNER]: Refusing update from remote peer 9d7c8c0f7907428eab22633702ead7ab: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240501 14:09:34.628613 8953 raft_consensus.cc:1261] T 6841e1b32b86454f9e40b64c141a2a6b P 9faf312806f54f2e8a4a807b234e0d1b [term 3 FOLLOWER]: Refusing update from remote peer 9d7c8c0f7907428eab22633702ead7ab: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240501 14:09:34.628888 11986 consensus_queue.cc:1035] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:09:34.629177 7904 raft_consensus.cc:1261] T 6841e1b32b86454f9e40b64c141a2a6b P 47eabc135e9f4237b036097085e729d8 [term 3 FOLLOWER]: Refusing update from remote peer 9d7c8c0f7907428eab22633702ead7ab: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240501 14:09:34.629576 11986 consensus_queue.cc:1035] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:09:34.629946 11988 consensus_queue.cc:1035] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:09:34.631961 11986 raft_consensus.cc:2932] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab [term 3 LEADER]: Committing config change with OpId 3.13: config changed from index 12 to 13, 9d32e843754e44d0b9aae6f472a4d9a2 (127.7.39.136) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } }
I20240501 14:09:34.632431 7904 raft_consensus.cc:2932] T 6841e1b32b86454f9e40b64c141a2a6b P 47eabc135e9f4237b036097085e729d8 [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 12 to 13, 9d32e843754e44d0b9aae6f472a4d9a2 (127.7.39.136) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } }
I20240501 14:09:34.632465 8952 raft_consensus.cc:2932] T 6841e1b32b86454f9e40b64c141a2a6b P 9faf312806f54f2e8a4a807b234e0d1b [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 12 to 13, 9d32e843754e44d0b9aae6f472a4d9a2 (127.7.39.136) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } }
I20240501 14:09:34.635186 9346 raft_consensus.cc:2932] T 6841e1b32b86454f9e40b64c141a2a6b P 9d32e843754e44d0b9aae6f472a4d9a2 [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 12 to 13, 9d32e843754e44d0b9aae6f472a4d9a2 (127.7.39.136) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } }
I20240501 14:09:34.634939 7365 catalog_manager.cc:5484] T 6841e1b32b86454f9e40b64c141a2a6b P 47eabc135e9f4237b036097085e729d8 reported cstate change: config changed from index 12 to 13, 9d32e843754e44d0b9aae6f472a4d9a2 (127.7.39.136) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "9d7c8c0f7907428eab22633702ead7ab" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } }
I20240501 14:09:34.639721 9083 consensus_queue.cc:237] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 3.13, Last appended by leader: 9, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } }
I20240501 14:09:34.640928 9346 raft_consensus.cc:1261] T 6841e1b32b86454f9e40b64c141a2a6b P 9d32e843754e44d0b9aae6f472a4d9a2 [term 3 FOLLOWER]: Refusing update from remote peer 9d7c8c0f7907428eab22633702ead7ab: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240501 14:09:34.640951 8952 raft_consensus.cc:1261] T 6841e1b32b86454f9e40b64c141a2a6b P 9faf312806f54f2e8a4a807b234e0d1b [term 3 FOLLOWER]: Refusing update from remote peer 9d7c8c0f7907428eab22633702ead7ab: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240501 14:09:34.641311 11988 consensus_queue.cc:1035] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 14:09:34.641489 11993 consensus_queue.cc:1035] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 14:09:34.642902 11988 raft_consensus.cc:2932] T 6841e1b32b86454f9e40b64c141a2a6b P 9d7c8c0f7907428eab22633702ead7ab [term 3 LEADER]: Committing config change with OpId 3.14: config changed from index 13 to 14, VOTER 47eabc135e9f4237b036097085e729d8 (127.7.39.132) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } }
I20240501 14:09:34.643330 9346 raft_consensus.cc:2932] T 6841e1b32b86454f9e40b64c141a2a6b P 9d32e843754e44d0b9aae6f472a4d9a2 [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 13 to 14, VOTER 47eabc135e9f4237b036097085e729d8 (127.7.39.132) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } }
I20240501 14:09:34.643496 8952 raft_consensus.cc:2932] T 6841e1b32b86454f9e40b64c141a2a6b P 9faf312806f54f2e8a4a807b234e0d1b [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 13 to 14, VOTER 47eabc135e9f4237b036097085e729d8 (127.7.39.132) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } }
I20240501 14:09:34.644322 7356 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6841e1b32b86454f9e40b64c141a2a6b with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:34.645697 7369 catalog_manager.cc:5484] T 6841e1b32b86454f9e40b64c141a2a6b P 9faf312806f54f2e8a4a807b234e0d1b reported cstate change: config changed from index 13 to 14, VOTER 47eabc135e9f4237b036097085e729d8 (127.7.39.132) evicted. New cstate: current_term: 3 leader_uuid: "9d7c8c0f7907428eab22633702ead7ab" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9d7c8c0f7907428eab22633702ead7ab" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 38699 } attrs { promote: false } } peers { permanent_uuid: "9faf312806f54f2e8a4a807b234e0d1b" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 38805 } attrs { promote: false } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: false } } }
I20240501 14:09:34.648026 7882 tablet_service.cc:1514] Processing DeleteTablet for tablet 6841e1b32b86454f9e40b64c141a2a6b with delete_type TABLET_DATA_TOMBSTONED (TS 47eabc135e9f4237b036097085e729d8 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:43764
I20240501 14:09:34.648509 11998 tablet_replica.cc:330] T 6841e1b32b86454f9e40b64c141a2a6b P 47eabc135e9f4237b036097085e729d8: stopping tablet replica
I20240501 14:09:34.648716 11998 raft_consensus.cc:2229] T 6841e1b32b86454f9e40b64c141a2a6b P 47eabc135e9f4237b036097085e729d8 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:34.648970 11998 raft_consensus.cc:2258] T 6841e1b32b86454f9e40b64c141a2a6b P 47eabc135e9f4237b036097085e729d8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.649675 11998 ts_tablet_manager.cc:1875] T 6841e1b32b86454f9e40b64c141a2a6b P 47eabc135e9f4237b036097085e729d8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:34.651535 11998 ts_tablet_manager.cc:1888] T 6841e1b32b86454f9e40b64c141a2a6b P 47eabc135e9f4237b036097085e729d8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.13
I20240501 14:09:34.651595 11998 log.cc:1196] T 6841e1b32b86454f9e40b64c141a2a6b P 47eabc135e9f4237b036097085e729d8: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal/wals/6841e1b32b86454f9e40b64c141a2a6b
I20240501 14:09:34.652041 7354 catalog_manager.cc:4830] TS 47eabc135e9f4237b036097085e729d8 (127.7.39.132:42369): tablet 6841e1b32b86454f9e40b64c141a2a6b (table rebalance_test_table_3 [id=1efd7e27dfbf4a88a4c2c2d4d7ebfe4f]) successfully deleted
I20240501 14:09:34.801362 9326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:34.802579 7752 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:34.803170 7620 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:34.804109 9064 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:34.811151 9195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:34.812014 8931 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:34.816524 7882 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:34.911440 7882 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:34.912997 9326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:34.915225 7752 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:34.917883 9064 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:34.918849 9195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:34.930696 7620 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:34.946749 8931 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:34.977666 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.129:37083
--local_ip_for_outbound_sockets=127.7.39.129
--tserver_master_addrs=127.7.39.190:44225
--webserver_port=42839
--webserver_interface=127.7.39.129
--builtin_ntp_servers=127.7.39.148:46693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240501 14:09:34.983453 7708 consensus_peers.cc:485] T dcf673c75a8b48da95806e1c43309dda P 9ae5a87ca07b40a788a42734b24f66c6 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.129:37083: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:09:35.089393 12075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:35.089633 12075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:35.089712 12075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:35.092610 12075 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:35.092680 12075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:35.092779 12075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.129
I20240501 14:09:35.096331 12075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:46693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.129:37083
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.7.39.129
--webserver_port=42839
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.7.39.190:44225
--never_fsync=true
--heap_profile_path=/tmp/kudu.12075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.129
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:35.097237 12075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:35.098443 12075 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:35.107510 12084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:35.108464 12082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:35.109843 12081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:35.110996 12075 server_base.cc:1034] running on GCE node
I20240501 14:09:35.111469 12075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:35.112063 12075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:35.113229 12075 hybrid_clock.cc:648] HybridClock initialized: now 1714572575113212 us; error 33 us; skew 500 ppm
I20240501 14:09:35.123472 12075 webserver.cc:460] Webserver started at http://127.7.39.129:42839/ using document root <none> and password file <none>
I20240501 14:09:35.124126 12075 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:35.124229 12075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:35.127993 12075 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20240501 14:09:35.130882 12090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:35.131897 12075 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:09:35.132016 12075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "ae52f888f5094416a24cb3dffbff03c4"
format_stamp: "Formatted at 2024-05-01 14:08:48 on dist-test-slave-rnfw"
I20240501 14:09:35.132453 12075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:35.147198 12075 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:35.147991 12075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:35.148172 12075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:35.148707 12075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:35.149855 12097 ts_tablet_manager.cc:516] Loading tablet metadata (0/12 complete)
I20240501 14:09:35.159909 12075 ts_tablet_manager.cc:559] Loaded tablet metadata (12 total tablets, 12 live tablets)
I20240501 14:09:35.159978 12075 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.010s user 0.001s sys 0.000s
I20240501 14:09:35.160043 12075 ts_tablet_manager.cc:574] Registering tablets (0/12 complete)
I20240501 14:09:35.162078 12097 tablet_bootstrap.cc:492] T 2cf2f85819584fdf872f6c590d11f873 P ae52f888f5094416a24cb3dffbff03c4: Bootstrap starting.
I20240501 14:09:35.166414 12075 ts_tablet_manager.cc:590] Registered 12 tablets
I20240501 14:09:35.166465 12075 ts_tablet_manager.cc:569] Time spent register tablets: real 0.006s user 0.004s sys 0.000s
I20240501 14:09:35.177273 12075 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.129:37083
I20240501 14:09:35.177464 12206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.129:37083 every 8 connection(s)
I20240501 14:09:35.177649 12097 log.cc:824] T 2cf2f85819584fdf872f6c590d11f873 P ae52f888f5094416a24cb3dffbff03c4: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:35.178511 12075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 14:09:35.184712 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 12075
I20240501 14:09:35.192137 12209 heartbeater.cc:347] Connected to a master server at 127.7.39.190:44225
I20240501 14:09:35.192404 12209 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:35.193033 12209 heartbeater.cc:511] Master 127.7.39.190:44225 requested a full tablet report, sending...
I20240501 14:09:35.195189 7366 ts_manager.cc:194] Re-registered known tserver with Master: ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083)
I20240501 14:09:35.197970 7366 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.129:41291
I20240501 14:09:35.208564 12097 tablet_bootstrap.cc:492] T 2cf2f85819584fdf872f6c590d11f873 P ae52f888f5094416a24cb3dffbff03c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:35.209055 12097 tablet_bootstrap.cc:492] T 2cf2f85819584fdf872f6c590d11f873 P ae52f888f5094416a24cb3dffbff03c4: Bootstrap complete.
I20240501 14:09:35.210395 12097 ts_tablet_manager.cc:1367] T 2cf2f85819584fdf872f6c590d11f873 P ae52f888f5094416a24cb3dffbff03c4: Time spent bootstrapping tablet: real 0.049s user 0.038s sys 0.008s
I20240501 14:09:35.212922 12097 raft_consensus.cc:348] T 2cf2f85819584fdf872f6c590d11f873 P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:09:35.213325 12097 raft_consensus.cc:729] T 2cf2f85819584fdf872f6c590d11f873 P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae52f888f5094416a24cb3dffbff03c4, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.213790 12097 consensus_queue.cc:260] T 2cf2f85819584fdf872f6c590d11f873 P ae52f888f5094416a24cb3dffbff03c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:09:35.214493 12209 heartbeater.cc:503] Master 127.7.39.190:44225 was elected leader, sending a full tablet report...
I20240501 14:09:35.214711 12097 ts_tablet_manager.cc:1398] T 2cf2f85819584fdf872f6c590d11f873 P ae52f888f5094416a24cb3dffbff03c4: Time spent starting tablet: real 0.004s user 0.007s sys 0.000s
I20240501 14:09:35.215009 12097 tablet_bootstrap.cc:492] T 3137e7904ad04fc79d4028c1882e2412 P ae52f888f5094416a24cb3dffbff03c4: Bootstrap starting.
W20240501 14:09:35.232910 7708 consensus_peers.cc:485] T ceee52d679bf4d8e8896a6c0dde0519b P 9ae5a87ca07b40a788a42734b24f66c6 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 71: this message will repeat every 5th retry.
I20240501 14:09:35.248550 12097 tablet_bootstrap.cc:492] T 3137e7904ad04fc79d4028c1882e2412 P ae52f888f5094416a24cb3dffbff03c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:35.249007 12097 tablet_bootstrap.cc:492] T 3137e7904ad04fc79d4028c1882e2412 P ae52f888f5094416a24cb3dffbff03c4: Bootstrap complete.
I20240501 14:09:35.250109 12097 ts_tablet_manager.cc:1367] T 3137e7904ad04fc79d4028c1882e2412 P ae52f888f5094416a24cb3dffbff03c4: Time spent bootstrapping tablet: real 0.035s user 0.031s sys 0.000s
I20240501 14:09:35.250700 12097 raft_consensus.cc:348] T 3137e7904ad04fc79d4028c1882e2412 P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } }
I20240501 14:09:35.250838 12097 raft_consensus.cc:729] T 3137e7904ad04fc79d4028c1882e2412 P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae52f888f5094416a24cb3dffbff03c4, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.250977 12097 consensus_queue.cc:260] T 3137e7904ad04fc79d4028c1882e2412 P ae52f888f5094416a24cb3dffbff03c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } }
I20240501 14:09:35.251272 12097 ts_tablet_manager.cc:1398] T 3137e7904ad04fc79d4028c1882e2412 P ae52f888f5094416a24cb3dffbff03c4: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 14:09:35.251510 12097 tablet_bootstrap.cc:492] T ab7c1f81f7524d55bd43065890088eaf P ae52f888f5094416a24cb3dffbff03c4: Bootstrap starting.
I20240501 14:09:35.274130 12139 tablet_service.cc:1514] Processing DeleteTablet for tablet c747bd6504c540cebded0019d82495b4 with delete_type TABLET_DATA_TOMBSTONED (TS ae52f888f5094416a24cb3dffbff03c4 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:49746
W20240501 14:09:35.275193 7355 catalog_manager.cc:4810] TS ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): delete failed for tablet c747bd6504c540cebded0019d82495b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c747bd6504c540cebded0019d82495b4 already in progress: opening tablet
I20240501 14:09:35.284335 12097 tablet_bootstrap.cc:492] T ab7c1f81f7524d55bd43065890088eaf P ae52f888f5094416a24cb3dffbff03c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:35.284756 12097 tablet_bootstrap.cc:492] T ab7c1f81f7524d55bd43065890088eaf P ae52f888f5094416a24cb3dffbff03c4: Bootstrap complete.
I20240501 14:09:35.285935 12097 ts_tablet_manager.cc:1367] T ab7c1f81f7524d55bd43065890088eaf P ae52f888f5094416a24cb3dffbff03c4: Time spent bootstrapping tablet: real 0.034s user 0.031s sys 0.000s
I20240501 14:09:35.286418 12097 raft_consensus.cc:348] T ab7c1f81f7524d55bd43065890088eaf P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:09:35.286548 12097 raft_consensus.cc:729] T ab7c1f81f7524d55bd43065890088eaf P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae52f888f5094416a24cb3dffbff03c4, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.286689 12097 consensus_queue.cc:260] T ab7c1f81f7524d55bd43065890088eaf P ae52f888f5094416a24cb3dffbff03c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:09:35.286993 12097 ts_tablet_manager.cc:1398] T ab7c1f81f7524d55bd43065890088eaf P ae52f888f5094416a24cb3dffbff03c4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:35.287197 12097 tablet_bootstrap.cc:492] T ae06bed9bd844a05ae2ac69148bf5bad P ae52f888f5094416a24cb3dffbff03c4: Bootstrap starting.
I20240501 14:09:35.319962 12097 tablet_bootstrap.cc:492] T ae06bed9bd844a05ae2ac69148bf5bad P ae52f888f5094416a24cb3dffbff03c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:35.320400 12097 tablet_bootstrap.cc:492] T ae06bed9bd844a05ae2ac69148bf5bad P ae52f888f5094416a24cb3dffbff03c4: Bootstrap complete.
I20240501 14:09:35.321547 12097 ts_tablet_manager.cc:1367] T ae06bed9bd844a05ae2ac69148bf5bad P ae52f888f5094416a24cb3dffbff03c4: Time spent bootstrapping tablet: real 0.034s user 0.023s sys 0.008s
I20240501 14:09:35.322012 12097 raft_consensus.cc:348] T ae06bed9bd844a05ae2ac69148bf5bad P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:09:35.322139 12097 raft_consensus.cc:729] T ae06bed9bd844a05ae2ac69148bf5bad P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae52f888f5094416a24cb3dffbff03c4, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.322340 12097 consensus_queue.cc:260] T ae06bed9bd844a05ae2ac69148bf5bad P ae52f888f5094416a24cb3dffbff03c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:09:35.322685 12097 ts_tablet_manager.cc:1398] T ae06bed9bd844a05ae2ac69148bf5bad P ae52f888f5094416a24cb3dffbff03c4: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240501 14:09:35.322858 12097 tablet_bootstrap.cc:492] T dcf673c75a8b48da95806e1c43309dda P ae52f888f5094416a24cb3dffbff03c4: Bootstrap starting.
I20240501 14:09:35.354583 12097 tablet_bootstrap.cc:492] T dcf673c75a8b48da95806e1c43309dda P ae52f888f5094416a24cb3dffbff03c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:35.355026 12097 tablet_bootstrap.cc:492] T dcf673c75a8b48da95806e1c43309dda P ae52f888f5094416a24cb3dffbff03c4: Bootstrap complete.
I20240501 14:09:35.356175 12097 ts_tablet_manager.cc:1367] T dcf673c75a8b48da95806e1c43309dda P ae52f888f5094416a24cb3dffbff03c4: Time spent bootstrapping tablet: real 0.033s user 0.028s sys 0.003s
I20240501 14:09:35.356650 12097 raft_consensus.cc:348] T dcf673c75a8b48da95806e1c43309dda P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:09:35.356776 12097 raft_consensus.cc:729] T dcf673c75a8b48da95806e1c43309dda P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae52f888f5094416a24cb3dffbff03c4, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.356920 12097 consensus_queue.cc:260] T dcf673c75a8b48da95806e1c43309dda P ae52f888f5094416a24cb3dffbff03c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:09:35.357267 12097 ts_tablet_manager.cc:1398] T dcf673c75a8b48da95806e1c43309dda P ae52f888f5094416a24cb3dffbff03c4: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:09:35.357496 12097 tablet_bootstrap.cc:492] T df856440b07b418394477cf368b765c1 P ae52f888f5094416a24cb3dffbff03c4: Bootstrap starting.
I20240501 14:09:35.382766 12097 tablet_bootstrap.cc:492] T df856440b07b418394477cf368b765c1 P ae52f888f5094416a24cb3dffbff03c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:35.383176 12097 tablet_bootstrap.cc:492] T df856440b07b418394477cf368b765c1 P ae52f888f5094416a24cb3dffbff03c4: Bootstrap complete.
I20240501 14:09:35.384179 12097 ts_tablet_manager.cc:1367] T df856440b07b418394477cf368b765c1 P ae52f888f5094416a24cb3dffbff03c4: Time spent bootstrapping tablet: real 0.027s user 0.018s sys 0.008s
I20240501 14:09:35.384696 12097 raft_consensus.cc:348] T df856440b07b418394477cf368b765c1 P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:09:35.384886 12097 raft_consensus.cc:729] T df856440b07b418394477cf368b765c1 P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae52f888f5094416a24cb3dffbff03c4, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.385054 12097 consensus_queue.cc:260] T df856440b07b418394477cf368b765c1 P ae52f888f5094416a24cb3dffbff03c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:09:35.385339 12097 ts_tablet_manager.cc:1398] T df856440b07b418394477cf368b765c1 P ae52f888f5094416a24cb3dffbff03c4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:35.385473 12097 tablet_bootstrap.cc:492] T 2880045de45743bdb5358a17d93b2245 P ae52f888f5094416a24cb3dffbff03c4: Bootstrap starting.
I20240501 14:09:35.414672 12097 tablet_bootstrap.cc:492] T 2880045de45743bdb5358a17d93b2245 P ae52f888f5094416a24cb3dffbff03c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:35.415230 12097 tablet_bootstrap.cc:492] T 2880045de45743bdb5358a17d93b2245 P ae52f888f5094416a24cb3dffbff03c4: Bootstrap complete.
I20240501 14:09:35.416297 12097 ts_tablet_manager.cc:1367] T 2880045de45743bdb5358a17d93b2245 P ae52f888f5094416a24cb3dffbff03c4: Time spent bootstrapping tablet: real 0.031s user 0.022s sys 0.008s
I20240501 14:09:35.416823 12097 raft_consensus.cc:348] T 2880045de45743bdb5358a17d93b2245 P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:09:35.416960 12097 raft_consensus.cc:729] T 2880045de45743bdb5358a17d93b2245 P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae52f888f5094416a24cb3dffbff03c4, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.417122 12097 consensus_queue.cc:260] T 2880045de45743bdb5358a17d93b2245 P ae52f888f5094416a24cb3dffbff03c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } }
I20240501 14:09:35.417428 12097 ts_tablet_manager.cc:1398] T 2880045de45743bdb5358a17d93b2245 P ae52f888f5094416a24cb3dffbff03c4: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:09:35.417645 12097 tablet_bootstrap.cc:492] T c747bd6504c540cebded0019d82495b4 P ae52f888f5094416a24cb3dffbff03c4: Bootstrap starting.
W20240501 14:09:35.436762 7839 consensus_peers.cc:485] T 4d282d79e1664f7ea8912e0c5166eec3 P 47eabc135e9f4237b036097085e729d8 -> Peer ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): Couldn't send request to peer ae52f888f5094416a24cb3dffbff03c4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 71: this message will repeat every 5th retry.
I20240501 14:09:35.437348 7882 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:35.445415 9064 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:35.447687 8931 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:35.454443 12139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:35.456048 12097 tablet_bootstrap.cc:492] T c747bd6504c540cebded0019d82495b4 P ae52f888f5094416a24cb3dffbff03c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:35.456476 12097 tablet_bootstrap.cc:492] T c747bd6504c540cebded0019d82495b4 P ae52f888f5094416a24cb3dffbff03c4: Bootstrap complete.
I20240501 14:09:35.457639 12097 ts_tablet_manager.cc:1367] T c747bd6504c540cebded0019d82495b4 P ae52f888f5094416a24cb3dffbff03c4: Time spent bootstrapping tablet: real 0.040s user 0.032s sys 0.004s
I20240501 14:09:35.458102 12097 raft_consensus.cc:348] T c747bd6504c540cebded0019d82495b4 P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: NON_VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: true } }
I20240501 14:09:35.458256 12097 raft_consensus.cc:729] T c747bd6504c540cebded0019d82495b4 P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae52f888f5094416a24cb3dffbff03c4, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.458395 12097 consensus_queue.cc:260] T c747bd6504c540cebded0019d82495b4 P ae52f888f5094416a24cb3dffbff03c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } attrs { replace: true } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "9d32e843754e44d0b9aae6f472a4d9a2" member_type: NON_VOTER last_known_addr { host: "127.7.39.136" port: 37475 } attrs { promote: true } }
I20240501 14:09:35.458750 12097 ts_tablet_manager.cc:1398] T c747bd6504c540cebded0019d82495b4 P ae52f888f5094416a24cb3dffbff03c4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:35.458928 12097 tablet_bootstrap.cc:492] T 4d282d79e1664f7ea8912e0c5166eec3 P ae52f888f5094416a24cb3dffbff03c4: Bootstrap starting.
I20240501 14:09:35.460536 12139 tablet_service.cc:1514] Processing DeleteTablet for tablet c747bd6504c540cebded0019d82495b4 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 5 (current committed config index is 15)) from {username='slave'} at 127.0.0.1:49746
I20240501 14:09:35.460904 12217 tablet_replica.cc:330] stopping tablet replica
I20240501 14:09:35.461220 12217 raft_consensus.cc:2229] T c747bd6504c540cebded0019d82495b4 P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:35.461508 12217 raft_consensus.cc:2258] T c747bd6504c540cebded0019d82495b4 P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.462623 12217 ts_tablet_manager.cc:1875] T c747bd6504c540cebded0019d82495b4 P ae52f888f5094416a24cb3dffbff03c4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:35.467582 12217 ts_tablet_manager.cc:1888] T c747bd6504c540cebded0019d82495b4 P ae52f888f5094416a24cb3dffbff03c4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:09:35.467777 12217 log.cc:1196] T c747bd6504c540cebded0019d82495b4 P ae52f888f5094416a24cb3dffbff03c4: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal/wals/c747bd6504c540cebded0019d82495b4
I20240501 14:09:35.468570 7355 catalog_manager.cc:4830] TS ae52f888f5094416a24cb3dffbff03c4 (127.7.39.129:37083): tablet c747bd6504c540cebded0019d82495b4 (table rebalance_test_table_4 [id=9b4ecb4938fd4bc185e20ce3ec5254f5]) successfully deleted
I20240501 14:09:35.471331 9195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:35.479802 9326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:35.498438 7752 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:35.498529 7620 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:35.503648 12097 tablet_bootstrap.cc:492] T 4d282d79e1664f7ea8912e0c5166eec3 P ae52f888f5094416a24cb3dffbff03c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:35.504109 12097 tablet_bootstrap.cc:492] T 4d282d79e1664f7ea8912e0c5166eec3 P ae52f888f5094416a24cb3dffbff03c4: Bootstrap complete.
I20240501 14:09:35.505278 12097 ts_tablet_manager.cc:1367] T 4d282d79e1664f7ea8912e0c5166eec3 P ae52f888f5094416a24cb3dffbff03c4: Time spent bootstrapping tablet: real 0.046s user 0.024s sys 0.008s
I20240501 14:09:35.505712 12097 raft_consensus.cc:348] T 4d282d79e1664f7ea8912e0c5166eec3 P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } }
I20240501 14:09:35.505837 12097 raft_consensus.cc:729] T 4d282d79e1664f7ea8912e0c5166eec3 P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae52f888f5094416a24cb3dffbff03c4, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.505961 12097 consensus_queue.cc:260] T 4d282d79e1664f7ea8912e0c5166eec3 P ae52f888f5094416a24cb3dffbff03c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } }
I20240501 14:09:35.506223 12097 ts_tablet_manager.cc:1398] T 4d282d79e1664f7ea8912e0c5166eec3 P ae52f888f5094416a24cb3dffbff03c4: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:09:35.506387 12097 tablet_bootstrap.cc:492] T 14e325d8d3184c9192b8cebafb414190 P ae52f888f5094416a24cb3dffbff03c4: Bootstrap starting.
I20240501 14:09:35.536976 12097 tablet_bootstrap.cc:492] T 14e325d8d3184c9192b8cebafb414190 P ae52f888f5094416a24cb3dffbff03c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:35.537353 12097 tablet_bootstrap.cc:492] T 14e325d8d3184c9192b8cebafb414190 P ae52f888f5094416a24cb3dffbff03c4: Bootstrap complete.
I20240501 14:09:35.538475 12097 ts_tablet_manager.cc:1367] T 14e325d8d3184c9192b8cebafb414190 P ae52f888f5094416a24cb3dffbff03c4: Time spent bootstrapping tablet: real 0.032s user 0.021s sys 0.008s
I20240501 14:09:35.539000 12097 raft_consensus.cc:348] T 14e325d8d3184c9192b8cebafb414190 P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } }
I20240501 14:09:35.539232 12097 raft_consensus.cc:729] T 14e325d8d3184c9192b8cebafb414190 P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae52f888f5094416a24cb3dffbff03c4, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.539376 12097 consensus_queue.cc:260] T 14e325d8d3184c9192b8cebafb414190 P ae52f888f5094416a24cb3dffbff03c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "9e44efa72e7d4421b284ae74d1cb9634" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 46433 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } }
I20240501 14:09:35.539837 12097 ts_tablet_manager.cc:1398] T 14e325d8d3184c9192b8cebafb414190 P ae52f888f5094416a24cb3dffbff03c4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:35.540017 12097 tablet_bootstrap.cc:492] T ceee52d679bf4d8e8896a6c0dde0519b P ae52f888f5094416a24cb3dffbff03c4: Bootstrap starting.
I20240501 14:09:35.567386 12097 tablet_bootstrap.cc:492] T ceee52d679bf4d8e8896a6c0dde0519b P ae52f888f5094416a24cb3dffbff03c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:35.567777 12097 tablet_bootstrap.cc:492] T ceee52d679bf4d8e8896a6c0dde0519b P ae52f888f5094416a24cb3dffbff03c4: Bootstrap complete.
I20240501 14:09:35.568755 12097 ts_tablet_manager.cc:1367] T ceee52d679bf4d8e8896a6c0dde0519b P ae52f888f5094416a24cb3dffbff03c4: Time spent bootstrapping tablet: real 0.029s user 0.019s sys 0.007s
I20240501 14:09:35.569134 12097 raft_consensus.cc:348] T ceee52d679bf4d8e8896a6c0dde0519b P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:09:35.569245 12097 raft_consensus.cc:729] T ceee52d679bf4d8e8896a6c0dde0519b P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae52f888f5094416a24cb3dffbff03c4, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.569406 12097 consensus_queue.cc:260] T ceee52d679bf4d8e8896a6c0dde0519b P ae52f888f5094416a24cb3dffbff03c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:09:35.569643 12097 ts_tablet_manager.cc:1398] T ceee52d679bf4d8e8896a6c0dde0519b P ae52f888f5094416a24cb3dffbff03c4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:35.569789 12097 tablet_bootstrap.cc:492] T 2112bf96dc3343678aa6f46bf2a3b1f2 P ae52f888f5094416a24cb3dffbff03c4: Bootstrap starting.
I20240501 14:09:35.603104 12097 tablet_bootstrap.cc:492] T 2112bf96dc3343678aa6f46bf2a3b1f2 P ae52f888f5094416a24cb3dffbff03c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:35.603677 12097 tablet_bootstrap.cc:492] T 2112bf96dc3343678aa6f46bf2a3b1f2 P ae52f888f5094416a24cb3dffbff03c4: Bootstrap complete.
I20240501 14:09:35.604905 12097 ts_tablet_manager.cc:1367] T 2112bf96dc3343678aa6f46bf2a3b1f2 P ae52f888f5094416a24cb3dffbff03c4: Time spent bootstrapping tablet: real 0.035s user 0.024s sys 0.008s
I20240501 14:09:35.605430 12097 raft_consensus.cc:348] T 2112bf96dc3343678aa6f46bf2a3b1f2 P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:09:35.605734 9326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:35.606194 7882 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:35.606391 12097 raft_consensus.cc:729] T 2112bf96dc3343678aa6f46bf2a3b1f2 P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae52f888f5094416a24cb3dffbff03c4, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.606629 12097 consensus_queue.cc:260] T 2112bf96dc3343678aa6f46bf2a3b1f2 P ae52f888f5094416a24cb3dffbff03c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eabc135e9f4237b036097085e729d8" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 42369 } } peers { permanent_uuid: "ae52f888f5094416a24cb3dffbff03c4" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37083 } } peers { permanent_uuid: "9ae5a87ca07b40a788a42734b24f66c6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 34697 } }
I20240501 14:09:35.607018 12097 ts_tablet_manager.cc:1398] T 2112bf96dc3343678aa6f46bf2a3b1f2 P ae52f888f5094416a24cb3dffbff03c4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:35.617105 12139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:35.617852 7752 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:35.619451 9064 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:35.633323 8931 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:35.638638 9195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:35.647207 7620 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:35.647989 12144 raft_consensus.cc:3037] T df856440b07b418394477cf368b765c1 P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:35.656477 12144 raft_consensus.cc:3037] T 2cf2f85819584fdf872f6c590d11f873 P ae52f888f5094416a24cb3dffbff03c4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:35.686688 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 12075
I20240501 14:09:35.693090 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 7560
I20240501 14:09:35.699888 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 7691
I20240501 14:09:35.706583 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 7822
I20240501 14:09:35.713302 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 8871
I20240501 14:09:35.719293 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 9002
I20240501 14:09:35.725638 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 9133
I20240501 14:09:35.731616 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 9264
I20240501 14:09:35.739005 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 7338
2024-05-01T14:09:35Z chronyd exiting
[ OK ] IgnoredTserverGoesDownDuringRebalancingTest.TserverDown (47645 ms)
[----------] 1 test from IgnoredTserverGoesDownDuringRebalancingTest (47645 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/0
I20240501 14:09:35.777403 7326 test_util.cc:268] Using random seed: 156377711
I20240501 14:09:35.778211 7326 ts_itest-base.cc:112] Starting cluster with:
I20240501 14:09:35.778249 7326 ts_itest-base.cc:113] --------------
I20240501 14:09:35.778270 7326 ts_itest-base.cc:114] 5 tablet servers
I20240501 14:09:35.778286 7326 ts_itest-base.cc:115] 3 replicas per TS
I20240501 14:09:35.778313 7326 ts_itest-base.cc:116] --------------
2024-05-01T14:09:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T14:09:35Z Disabled control of system clock
I20240501 14:09:35.786808 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.39.190:34995
--webserver_interface=127.7.39.190
--webserver_port=0
--builtin_ntp_servers=127.7.39.148:39291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.39.190:34995
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240501 14:09:35.891991 12324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:35.892235 12324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:35.892307 12324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:35.895689 12324 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:09:35.895769 12324 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 14:09:35.895805 12324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:35.895855 12324 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 14:09:35.895898 12324 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 14:09:35.899502 12324 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:39291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.7.39.190:34995
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.39.190:34995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.7.39.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.12324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:35.900396 12324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:35.901499 12324 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:35.909585 12330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:35.910444 12332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:35.911836 12329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:35.912963 12324 server_base.cc:1034] running on GCE node
I20240501 14:09:35.913496 12324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:35.914456 12324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:35.915611 12324 hybrid_clock.cc:648] HybridClock initialized: now 1714572575915597 us; error 30 us; skew 500 ppm
I20240501 14:09:35.917467 12324 webserver.cc:460] Webserver started at http://127.7.39.190:46707/ using document root <none> and password file <none>
I20240501 14:09:35.918011 12324 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:35.918084 12324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:35.918278 12324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:35.920046 12324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "7b3558c2788e417fb1b6be22c03ec672"
format_stamp: "Formatted at 2024-05-01 14:09:35 on dist-test-slave-rnfw"
I20240501 14:09:35.920537 12324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "7b3558c2788e417fb1b6be22c03ec672"
format_stamp: "Formatted at 2024-05-01 14:09:35 on dist-test-slave-rnfw"
I20240501 14:09:35.923835 12324 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:09:35.926090 12338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:35.927106 12324 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:09:35.927222 12324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/wal
uuid: "7b3558c2788e417fb1b6be22c03ec672"
format_stamp: "Formatted at 2024-05-01 14:09:35 on dist-test-slave-rnfw"
I20240501 14:09:35.927306 12324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:35.953897 12324 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:35.954609 12324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:35.954794 12324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:35.961908 12324 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.190:34995
I20240501 14:09:35.961925 12390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.190:34995 every 8 connection(s)
I20240501 14:09:35.963006 12324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 14:09:35.965579 12391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.970530 12391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b3558c2788e417fb1b6be22c03ec672: Bootstrap starting.
I20240501 14:09:35.972450 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 12324
I20240501 14:09:35.972560 7326 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 14:09:35.972965 12391 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b3558c2788e417fb1b6be22c03ec672: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.973737 12391 log.cc:824] T 00000000000000000000000000000000 P 7b3558c2788e417fb1b6be22c03ec672: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:35.975791 12391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b3558c2788e417fb1b6be22c03ec672: No bootstrap required, opened a new log
I20240501 14:09:35.978772 12391 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b3558c2788e417fb1b6be22c03ec672 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b3558c2788e417fb1b6be22c03ec672" member_type: VOTER last_known_addr { host: "127.7.39.190" port: 34995 } }
I20240501 14:09:35.978983 12391 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b3558c2788e417fb1b6be22c03ec672 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.979099 12391 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b3558c2788e417fb1b6be22c03ec672 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b3558c2788e417fb1b6be22c03ec672, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.979517 12391 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b3558c2788e417fb1b6be22c03ec672 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b3558c2788e417fb1b6be22c03ec672" member_type: VOTER last_known_addr { host: "127.7.39.190" port: 34995 } }
I20240501 14:09:35.979643 12391 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b3558c2788e417fb1b6be22c03ec672 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:09:35.979708 12391 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b3558c2788e417fb1b6be22c03ec672 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:09:35.979815 12391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b3558c2788e417fb1b6be22c03ec672 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:35.980630 12391 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b3558c2788e417fb1b6be22c03ec672 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b3558c2788e417fb1b6be22c03ec672" member_type: VOTER last_known_addr { host: "127.7.39.190" port: 34995 } }
I20240501 14:09:35.980998 12391 leader_election.cc:302] T 00000000000000000000000000000000 P 7b3558c2788e417fb1b6be22c03ec672 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b3558c2788e417fb1b6be22c03ec672; no voters:
I20240501 14:09:35.981287 12391 leader_election.cc:288] T 00000000000000000000000000000000 P 7b3558c2788e417fb1b6be22c03ec672 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:09:35.981429 12396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b3558c2788e417fb1b6be22c03ec672 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:35.981685 12396 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b3558c2788e417fb1b6be22c03ec672 [term 1 LEADER]: Becoming Leader. State: Replica: 7b3558c2788e417fb1b6be22c03ec672, State: Running, Role: LEADER
I20240501 14:09:35.982087 12396 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b3558c2788e417fb1b6be22c03ec672 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b3558c2788e417fb1b6be22c03ec672" member_type: VOTER last_known_addr { host: "127.7.39.190" port: 34995 } }
I20240501 14:09:35.982436 12391 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b3558c2788e417fb1b6be22c03ec672 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:09:35.983449 12397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b3558c2788e417fb1b6be22c03ec672 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b3558c2788e417fb1b6be22c03ec672. Latest consensus state: current_term: 1 leader_uuid: "7b3558c2788e417fb1b6be22c03ec672" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b3558c2788e417fb1b6be22c03ec672" member_type: VOTER last_known_addr { host: "127.7.39.190" port: 34995 } } }
I20240501 14:09:35.983567 12397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b3558c2788e417fb1b6be22c03ec672 [sys.catalog]: This master's current role is: LEADER
I20240501 14:09:35.983855 12398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b3558c2788e417fb1b6be22c03ec672 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b3558c2788e417fb1b6be22c03ec672" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b3558c2788e417fb1b6be22c03ec672" member_type: VOTER last_known_addr { host: "127.7.39.190" port: 34995 } } }
I20240501 14:09:35.983932 12405 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:09:35.983984 12398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b3558c2788e417fb1b6be22c03ec672 [sys.catalog]: This master's current role is: LEADER
I20240501 14:09:35.986496 12405 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:09:35.991127 12405 catalog_manager.cc:1347] Generated new cluster ID: 814951363cc448c391a9bc473ee1932e
I20240501 14:09:35.991204 12405 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:09:36.012264 12405 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 14:09:36.013249 12405 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:09:36.022446 12405 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 7b3558c2788e417fb1b6be22c03ec672: Generated new TSK 0
I20240501 14:09:36.023216 12405 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:09:36.026631 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.129:0
--local_ip_for_outbound_sockets=127.7.39.129
--webserver_interface=127.7.39.129
--webserver_port=0
--tserver_master_addrs=127.7.39.190:34995
--builtin_ntp_servers=127.7.39.148:39291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 14:09:36.134752 12415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:36.135000 12415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:36.135085 12415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:36.138033 12415 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:09:36.138105 12415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:36.138175 12415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.129
I20240501 14:09:36.141613 12415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:39291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.7.39.129
--webserver_port=0
--tserver_master_addrs=127.7.39.190:34995
--never_fsync=true
--heap_profile_path=/tmp/kudu.12415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.129
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:36.142551 12415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:36.143726 12415 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:36.152597 12420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:36.153471 12421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:36.154832 12423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:36.155961 12415 server_base.cc:1034] running on GCE node
I20240501 14:09:36.156406 12415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:36.156980 12415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:36.158154 12415 hybrid_clock.cc:648] HybridClock initialized: now 1714572576158144 us; error 32 us; skew 500 ppm
I20240501 14:09:36.160120 12415 webserver.cc:460] Webserver started at http://127.7.39.129:42141/ using document root <none> and password file <none>
I20240501 14:09:36.160674 12415 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:36.160735 12415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:36.160907 12415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:36.162617 12415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "cf193f4215f74ad3959abcba3abf5ccc"
format_stamp: "Formatted at 2024-05-01 14:09:36 on dist-test-slave-rnfw"
I20240501 14:09:36.163180 12415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "cf193f4215f74ad3959abcba3abf5ccc"
format_stamp: "Formatted at 2024-05-01 14:09:36 on dist-test-slave-rnfw"
I20240501 14:09:36.166428 12415 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:36.168684 12429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:36.169675 12415 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:36.169786 12415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "cf193f4215f74ad3959abcba3abf5ccc"
format_stamp: "Formatted at 2024-05-01 14:09:36 on dist-test-slave-rnfw"
I20240501 14:09:36.169873 12415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:36.184845 12415 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:36.185405 12415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:36.185567 12415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:36.186074 12415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:36.186937 12415 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:36.186990 12415 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:36.187067 12415 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:36.187098 12415 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:36.195497 12415 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.129:41839
I20240501 14:09:36.195536 12542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.129:41839 every 8 connection(s)
I20240501 14:09:36.196435 12415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 14:09:36.203373 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 12415
I20240501 14:09:36.203510 7326 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 14:09:36.204926 12543 heartbeater.cc:347] Connected to a master server at 127.7.39.190:34995
I20240501 14:09:36.205166 12543 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:36.205595 12543 heartbeater.cc:511] Master 127.7.39.190:34995 requested a full tablet report, sending...
I20240501 14:09:36.205751 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.130:0
--local_ip_for_outbound_sockets=127.7.39.130
--webserver_interface=127.7.39.130
--webserver_port=0
--tserver_master_addrs=127.7.39.190:34995
--builtin_ntp_servers=127.7.39.148:39291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:09:36.207186 12354 ts_manager.cc:194] Registered new tserver with Master: cf193f4215f74ad3959abcba3abf5ccc (127.7.39.129:41839)
I20240501 14:09:36.208392 12354 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.129:58621
W20240501 14:09:36.314963 12546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:36.315217 12546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:36.315285 12546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:36.318120 12546 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:09:36.318187 12546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:36.318276 12546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.130
I20240501 14:09:36.321550 12546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:39291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.7.39.130
--webserver_port=0
--tserver_master_addrs=127.7.39.190:34995
--never_fsync=true
--heap_profile_path=/tmp/kudu.12546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.130
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:36.322423 12546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:36.323546 12546 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:36.332413 12554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:36.333680 12551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:36.334735 12552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:36.335999 12546 server_base.cc:1034] running on GCE node
I20240501 14:09:36.336374 12546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:36.336933 12546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:36.338099 12546 hybrid_clock.cc:648] HybridClock initialized: now 1714572576338086 us; error 28 us; skew 500 ppm
I20240501 14:09:36.340014 12546 webserver.cc:460] Webserver started at http://127.7.39.130:40753/ using document root <none> and password file <none>
I20240501 14:09:36.340689 12546 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:36.340768 12546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:36.340999 12546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:36.342635 12546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "04598e08a4654b7684ee8b47665ae4ac"
format_stamp: "Formatted at 2024-05-01 14:09:36 on dist-test-slave-rnfw"
I20240501 14:09:36.343117 12546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "04598e08a4654b7684ee8b47665ae4ac"
format_stamp: "Formatted at 2024-05-01 14:09:36 on dist-test-slave-rnfw"
I20240501 14:09:36.346319 12546 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:36.348459 12560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:36.349519 12546 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:36.349627 12546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "04598e08a4654b7684ee8b47665ae4ac"
format_stamp: "Formatted at 2024-05-01 14:09:36 on dist-test-slave-rnfw"
I20240501 14:09:36.349714 12546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:36.373098 12546 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:36.373788 12546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:36.373956 12546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:36.374532 12546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:36.375406 12546 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:36.375452 12546 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:36.375497 12546 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:36.375531 12546 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:36.384001 12546 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.130:37237
I20240501 14:09:36.384034 12673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.130:37237 every 8 connection(s)
I20240501 14:09:36.384917 12546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 14:09:36.392401 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 12546
I20240501 14:09:36.392547 7326 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 14:09:36.393386 12674 heartbeater.cc:347] Connected to a master server at 127.7.39.190:34995
I20240501 14:09:36.393611 12674 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:36.394116 12674 heartbeater.cc:511] Master 127.7.39.190:34995 requested a full tablet report, sending...
I20240501 14:09:36.394668 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.131:0
--local_ip_for_outbound_sockets=127.7.39.131
--webserver_interface=127.7.39.131
--webserver_port=0
--tserver_master_addrs=127.7.39.190:34995
--builtin_ntp_servers=127.7.39.148:39291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:09:36.395149 12354 ts_manager.cc:194] Registered new tserver with Master: 04598e08a4654b7684ee8b47665ae4ac (127.7.39.130:37237)
I20240501 14:09:36.395694 12354 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.130:42257
W20240501 14:09:36.499727 12677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:36.499954 12677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:36.500026 12677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:36.502745 12677 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:09:36.502815 12677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:36.502897 12677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.131
I20240501 14:09:36.506204 12677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:39291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.7.39.131
--webserver_port=0
--tserver_master_addrs=127.7.39.190:34995
--never_fsync=true
--heap_profile_path=/tmp/kudu.12677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.131
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:36.507098 12677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:36.508206 12677 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:36.517148 12685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:36.518445 12683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:36.519493 12682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:36.520757 12677 server_base.cc:1034] running on GCE node
I20240501 14:09:36.521160 12677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:36.521720 12677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:36.522886 12677 hybrid_clock.cc:648] HybridClock initialized: now 1714572576522874 us; error 27 us; skew 500 ppm
I20240501 14:09:36.524811 12677 webserver.cc:460] Webserver started at http://127.7.39.131:44003/ using document root <none> and password file <none>
I20240501 14:09:36.525323 12677 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:36.525377 12677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:36.525535 12677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:36.527189 12677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "70fab6e24a8f4173bc4dbed6ddf9e8a6"
format_stamp: "Formatted at 2024-05-01 14:09:36 on dist-test-slave-rnfw"
I20240501 14:09:36.527624 12677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "70fab6e24a8f4173bc4dbed6ddf9e8a6"
format_stamp: "Formatted at 2024-05-01 14:09:36 on dist-test-slave-rnfw"
I20240501 14:09:36.530774 12677 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:09:36.532992 12691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:36.533924 12677 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:36.534022 12677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "70fab6e24a8f4173bc4dbed6ddf9e8a6"
format_stamp: "Formatted at 2024-05-01 14:09:36 on dist-test-slave-rnfw"
I20240501 14:09:36.534119 12677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:36.553208 12677 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:36.553835 12677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:36.554039 12677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:36.554562 12677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:36.555482 12677 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:36.555529 12677 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:36.555569 12677 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:36.555604 12677 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:36.563764 12677 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.131:39579
I20240501 14:09:36.563793 12804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.131:39579 every 8 connection(s)
I20240501 14:09:36.564652 12677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 14:09:36.571494 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 12677
I20240501 14:09:36.571633 7326 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 14:09:36.573311 12805 heartbeater.cc:347] Connected to a master server at 127.7.39.190:34995
I20240501 14:09:36.573537 12805 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:36.573765 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.132:0
--local_ip_for_outbound_sockets=127.7.39.132
--webserver_interface=127.7.39.132
--webserver_port=0
--tserver_master_addrs=127.7.39.190:34995
--builtin_ntp_servers=127.7.39.148:39291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:09:36.574060 12805 heartbeater.cc:511] Master 127.7.39.190:34995 requested a full tablet report, sending...
I20240501 14:09:36.574950 12355 ts_manager.cc:194] Registered new tserver with Master: 70fab6e24a8f4173bc4dbed6ddf9e8a6 (127.7.39.131:39579)
I20240501 14:09:36.575511 12355 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.131:53163
W20240501 14:09:36.678678 12808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:36.678929 12808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:36.678987 12808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:36.681846 12808 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:09:36.681910 12808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:36.682014 12808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.132
I20240501 14:09:36.685497 12808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:39291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.7.39.132
--webserver_port=0
--tserver_master_addrs=127.7.39.190:34995
--never_fsync=true
--heap_profile_path=/tmp/kudu.12808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.132
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:36.686326 12808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:36.687415 12808 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:36.696022 12813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:36.697335 12814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:36.698422 12816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:36.699640 12808 server_base.cc:1034] running on GCE node
I20240501 14:09:36.700014 12808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:36.700556 12808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:36.701714 12808 hybrid_clock.cc:648] HybridClock initialized: now 1714572576701708 us; error 31 us; skew 500 ppm
I20240501 14:09:36.703619 12808 webserver.cc:460] Webserver started at http://127.7.39.132:46507/ using document root <none> and password file <none>
I20240501 14:09:36.704149 12808 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:36.704210 12808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:36.704378 12808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:36.705986 12808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "2b45210d2d4046b29e9e6114a6c96098"
format_stamp: "Formatted at 2024-05-01 14:09:36 on dist-test-slave-rnfw"
I20240501 14:09:36.706445 12808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "2b45210d2d4046b29e9e6114a6c96098"
format_stamp: "Formatted at 2024-05-01 14:09:36 on dist-test-slave-rnfw"
I20240501 14:09:36.709810 12808 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 14:09:36.711951 12822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:36.712891 12808 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:09:36.712996 12808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "2b45210d2d4046b29e9e6114a6c96098"
format_stamp: "Formatted at 2024-05-01 14:09:36 on dist-test-slave-rnfw"
I20240501 14:09:36.713079 12808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:36.729673 12808 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:36.730240 12808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:36.730396 12808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:36.730938 12808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:36.731904 12808 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:36.731966 12808 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:36.732015 12808 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:36.732036 12808 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:36.740454 12808 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.132:34297
I20240501 14:09:36.740460 12935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.132:34297 every 8 connection(s)
I20240501 14:09:36.741366 12808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 14:09:36.749430 12936 heartbeater.cc:347] Connected to a master server at 127.7.39.190:34995
I20240501 14:09:36.749666 12936 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:36.750056 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 12808
I20240501 14:09:36.750170 7326 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 14:09:36.750258 12936 heartbeater.cc:511] Master 127.7.39.190:34995 requested a full tablet report, sending...
I20240501 14:09:36.751613 12355 ts_manager.cc:194] Registered new tserver with Master: 2b45210d2d4046b29e9e6114a6c96098 (127.7.39.132:34297)
I20240501 14:09:36.752156 12355 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.132:58651
I20240501 14:09:36.752241 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.133:0
--local_ip_for_outbound_sockets=127.7.39.133
--webserver_interface=127.7.39.133
--webserver_port=0
--tserver_master_addrs=127.7.39.190:34995
--builtin_ntp_servers=127.7.39.148:39291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 14:09:36.857854 12939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:36.858088 12939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:36.858161 12939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:36.861127 12939 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:09:36.861202 12939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:36.861294 12939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.133
I20240501 14:09:36.864874 12939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:39291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.7.39.133
--webserver_port=0
--tserver_master_addrs=127.7.39.190:34995
--never_fsync=true
--heap_profile_path=/tmp/kudu.12939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.133
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:36.865759 12939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:36.866847 12939 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:36.876030 12945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:36.877290 12944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:36.878279 12947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:36.879489 12939 server_base.cc:1034] running on GCE node
I20240501 14:09:36.879913 12939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:36.880484 12939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:36.881633 12939 hybrid_clock.cc:648] HybridClock initialized: now 1714572576881625 us; error 28 us; skew 500 ppm
I20240501 14:09:36.883719 12939 webserver.cc:460] Webserver started at http://127.7.39.133:43469/ using document root <none> and password file <none>
I20240501 14:09:36.884255 12939 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:36.884310 12939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:36.884501 12939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:36.886132 12939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "513a87b9102f402187ec511b32f1b3eb"
format_stamp: "Formatted at 2024-05-01 14:09:36 on dist-test-slave-rnfw"
I20240501 14:09:36.886581 12939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "513a87b9102f402187ec511b32f1b3eb"
format_stamp: "Formatted at 2024-05-01 14:09:36 on dist-test-slave-rnfw"
I20240501 14:09:36.889777 12939 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 14:09:36.891887 12953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:36.892952 12939 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:36.893065 12939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "513a87b9102f402187ec511b32f1b3eb"
format_stamp: "Formatted at 2024-05-01 14:09:36 on dist-test-slave-rnfw"
I20240501 14:09:36.893141 12939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:36.919615 12939 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:36.920202 12939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:36.920336 12939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:36.920841 12939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:36.921691 12939 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:36.921738 12939 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:36.921777 12939 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:36.921797 12939 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:36.930032 12939 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.133:33811
I20240501 14:09:36.930052 13066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.133:33811 every 8 connection(s)
I20240501 14:09:36.930969 12939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 14:09:36.939308 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 12939
I20240501 14:09:36.939436 7326 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 14:09:36.939811 13067 heartbeater.cc:347] Connected to a master server at 127.7.39.190:34995
I20240501 14:09:36.940076 13067 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:36.940586 13067 heartbeater.cc:511] Master 127.7.39.190:34995 requested a full tablet report, sending...
I20240501 14:09:36.941582 12354 ts_manager.cc:194] Registered new tserver with Master: 513a87b9102f402187ec511b32f1b3eb (127.7.39.133:33811)
I20240501 14:09:36.942135 12354 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.133:45937
I20240501 14:09:36.953651 7326 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 14:09:36.962257 12354 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48404:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 14:09:36.980579 12870 tablet_service.cc:1467] Processing CreateTablet for tablet 979ab8d0c9434e7dbfbc19028a6373f9 (DEFAULT_TABLE table=TestTable [id=dd691d9b56c84650b8baa86ecbca7790]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:36.980656 13001 tablet_service.cc:1467] Processing CreateTablet for tablet 979ab8d0c9434e7dbfbc19028a6373f9 (DEFAULT_TABLE table=TestTable [id=dd691d9b56c84650b8baa86ecbca7790]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:36.980985 12739 tablet_service.cc:1467] Processing CreateTablet for tablet 979ab8d0c9434e7dbfbc19028a6373f9 (DEFAULT_TABLE table=TestTable [id=dd691d9b56c84650b8baa86ecbca7790]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:36.981484 12870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 979ab8d0c9434e7dbfbc19028a6373f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.981527 13001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 979ab8d0c9434e7dbfbc19028a6373f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.981712 12739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 979ab8d0c9434e7dbfbc19028a6373f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.986418 13084 tablet_bootstrap.cc:492] T 979ab8d0c9434e7dbfbc19028a6373f9 P 2b45210d2d4046b29e9e6114a6c96098: Bootstrap starting.
I20240501 14:09:36.987645 13085 tablet_bootstrap.cc:492] T 979ab8d0c9434e7dbfbc19028a6373f9 P 513a87b9102f402187ec511b32f1b3eb: Bootstrap starting.
I20240501 14:09:36.988427 13086 tablet_bootstrap.cc:492] T 979ab8d0c9434e7dbfbc19028a6373f9 P 70fab6e24a8f4173bc4dbed6ddf9e8a6: Bootstrap starting.
I20240501 14:09:36.988746 13084 tablet_bootstrap.cc:654] T 979ab8d0c9434e7dbfbc19028a6373f9 P 2b45210d2d4046b29e9e6114a6c96098: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:36.989476 13085 tablet_bootstrap.cc:654] T 979ab8d0c9434e7dbfbc19028a6373f9 P 513a87b9102f402187ec511b32f1b3eb: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:36.989539 13084 log.cc:824] T 979ab8d0c9434e7dbfbc19028a6373f9 P 2b45210d2d4046b29e9e6114a6c96098: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:36.989923 13086 tablet_bootstrap.cc:654] T 979ab8d0c9434e7dbfbc19028a6373f9 P 70fab6e24a8f4173bc4dbed6ddf9e8a6: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:36.990183 13085 log.cc:824] T 979ab8d0c9434e7dbfbc19028a6373f9 P 513a87b9102f402187ec511b32f1b3eb: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:36.990605 13086 log.cc:824] T 979ab8d0c9434e7dbfbc19028a6373f9 P 70fab6e24a8f4173bc4dbed6ddf9e8a6: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:36.991355 13084 tablet_bootstrap.cc:492] T 979ab8d0c9434e7dbfbc19028a6373f9 P 2b45210d2d4046b29e9e6114a6c96098: No bootstrap required, opened a new log
I20240501 14:09:36.991529 13084 ts_tablet_manager.cc:1367] T 979ab8d0c9434e7dbfbc19028a6373f9 P 2b45210d2d4046b29e9e6114a6c96098: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 14:09:36.991758 13085 tablet_bootstrap.cc:492] T 979ab8d0c9434e7dbfbc19028a6373f9 P 513a87b9102f402187ec511b32f1b3eb: No bootstrap required, opened a new log
I20240501 14:09:36.991935 13085 ts_tablet_manager.cc:1367] T 979ab8d0c9434e7dbfbc19028a6373f9 P 513a87b9102f402187ec511b32f1b3eb: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:09:36.992077 13086 tablet_bootstrap.cc:492] T 979ab8d0c9434e7dbfbc19028a6373f9 P 70fab6e24a8f4173bc4dbed6ddf9e8a6: No bootstrap required, opened a new log
I20240501 14:09:36.992225 13086 ts_tablet_manager.cc:1367] T 979ab8d0c9434e7dbfbc19028a6373f9 P 70fab6e24a8f4173bc4dbed6ddf9e8a6: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:09:36.994911 13086 raft_consensus.cc:348] T 979ab8d0c9434e7dbfbc19028a6373f9 P 70fab6e24a8f4173bc4dbed6ddf9e8a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70fab6e24a8f4173bc4dbed6ddf9e8a6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 39579 } } peers { permanent_uuid: "2b45210d2d4046b29e9e6114a6c96098" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 34297 } } peers { permanent_uuid: "513a87b9102f402187ec511b32f1b3eb" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 33811 } }
I20240501 14:09:36.995139 13086 raft_consensus.cc:374] T 979ab8d0c9434e7dbfbc19028a6373f9 P 70fab6e24a8f4173bc4dbed6ddf9e8a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:36.995196 13086 raft_consensus.cc:729] T 979ab8d0c9434e7dbfbc19028a6373f9 P 70fab6e24a8f4173bc4dbed6ddf9e8a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70fab6e24a8f4173bc4dbed6ddf9e8a6, State: Initialized, Role: FOLLOWER
I20240501 14:09:36.995254 13084 raft_consensus.cc:348] T 979ab8d0c9434e7dbfbc19028a6373f9 P 2b45210d2d4046b29e9e6114a6c96098 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70fab6e24a8f4173bc4dbed6ddf9e8a6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 39579 } } peers { permanent_uuid: "2b45210d2d4046b29e9e6114a6c96098" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 34297 } } peers { permanent_uuid: "513a87b9102f402187ec511b32f1b3eb" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 33811 } }
I20240501 14:09:36.995254 13085 raft_consensus.cc:348] T 979ab8d0c9434e7dbfbc19028a6373f9 P 513a87b9102f402187ec511b32f1b3eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70fab6e24a8f4173bc4dbed6ddf9e8a6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 39579 } } peers { permanent_uuid: "2b45210d2d4046b29e9e6114a6c96098" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 34297 } } peers { permanent_uuid: "513a87b9102f402187ec511b32f1b3eb" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 33811 } }
I20240501 14:09:36.995440 13085 raft_consensus.cc:374] T 979ab8d0c9434e7dbfbc19028a6373f9 P 513a87b9102f402187ec511b32f1b3eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:36.995440 13084 raft_consensus.cc:374] T 979ab8d0c9434e7dbfbc19028a6373f9 P 2b45210d2d4046b29e9e6114a6c96098 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:36.995501 13085 raft_consensus.cc:729] T 979ab8d0c9434e7dbfbc19028a6373f9 P 513a87b9102f402187ec511b32f1b3eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 513a87b9102f402187ec511b32f1b3eb, State: Initialized, Role: FOLLOWER
I20240501 14:09:36.995501 13084 raft_consensus.cc:729] T 979ab8d0c9434e7dbfbc19028a6373f9 P 2b45210d2d4046b29e9e6114a6c96098 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b45210d2d4046b29e9e6114a6c96098, State: Initialized, Role: FOLLOWER
I20240501 14:09:36.995606 13086 consensus_queue.cc:260] T 979ab8d0c9434e7dbfbc19028a6373f9 P 70fab6e24a8f4173bc4dbed6ddf9e8a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70fab6e24a8f4173bc4dbed6ddf9e8a6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 39579 } } peers { permanent_uuid: "2b45210d2d4046b29e9e6114a6c96098" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 34297 } } peers { permanent_uuid: "513a87b9102f402187ec511b32f1b3eb" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 33811 } }
I20240501 14:09:36.995923 13085 consensus_queue.cc:260] T 979ab8d0c9434e7dbfbc19028a6373f9 P 513a87b9102f402187ec511b32f1b3eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70fab6e24a8f4173bc4dbed6ddf9e8a6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 39579 } } peers { permanent_uuid: "2b45210d2d4046b29e9e6114a6c96098" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 34297 } } peers { permanent_uuid: "513a87b9102f402187ec511b32f1b3eb" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 33811 } }
I20240501 14:09:36.995937 13084 consensus_queue.cc:260] T 979ab8d0c9434e7dbfbc19028a6373f9 P 2b45210d2d4046b29e9e6114a6c96098 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70fab6e24a8f4173bc4dbed6ddf9e8a6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 39579 } } peers { permanent_uuid: "2b45210d2d4046b29e9e6114a6c96098" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 34297 } } peers { permanent_uuid: "513a87b9102f402187ec511b32f1b3eb" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 33811 } }
I20240501 14:09:36.996457 13086 ts_tablet_manager.cc:1398] T 979ab8d0c9434e7dbfbc19028a6373f9 P 70fab6e24a8f4173bc4dbed6ddf9e8a6: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240501 14:09:36.996686 13084 ts_tablet_manager.cc:1398] T 979ab8d0c9434e7dbfbc19028a6373f9 P 2b45210d2d4046b29e9e6114a6c96098: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 14:09:36.996686 13085 ts_tablet_manager.cc:1398] T 979ab8d0c9434e7dbfbc19028a6373f9 P 513a87b9102f402187ec511b32f1b3eb: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 14:09:36.996832 12805 heartbeater.cc:503] Master 127.7.39.190:34995 was elected leader, sending a full tablet report...
I20240501 14:09:36.996861 13067 heartbeater.cc:503] Master 127.7.39.190:34995 was elected leader, sending a full tablet report...
I20240501 14:09:36.998247 12936 heartbeater.cc:503] Master 127.7.39.190:34995 was elected leader, sending a full tablet report...
I20240501 14:09:37.020275 13091 raft_consensus.cc:482] T 979ab8d0c9434e7dbfbc19028a6373f9 P 2b45210d2d4046b29e9e6114a6c96098 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:37.020465 13091 raft_consensus.cc:504] T 979ab8d0c9434e7dbfbc19028a6373f9 P 2b45210d2d4046b29e9e6114a6c96098 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70fab6e24a8f4173bc4dbed6ddf9e8a6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 39579 } } peers { permanent_uuid: "2b45210d2d4046b29e9e6114a6c96098" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 34297 } } peers { permanent_uuid: "513a87b9102f402187ec511b32f1b3eb" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 33811 } }
I20240501 14:09:37.021435 13091 leader_election.cc:288] T 979ab8d0c9434e7dbfbc19028a6373f9 P 2b45210d2d4046b29e9e6114a6c96098 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70fab6e24a8f4173bc4dbed6ddf9e8a6 (127.7.39.131:39579), 513a87b9102f402187ec511b32f1b3eb (127.7.39.133:33811)
I20240501 14:09:37.025489 12759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "979ab8d0c9434e7dbfbc19028a6373f9" candidate_uuid: "2b45210d2d4046b29e9e6114a6c96098" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70fab6e24a8f4173bc4dbed6ddf9e8a6" is_pre_election: true
I20240501 14:09:37.025601 13021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "979ab8d0c9434e7dbfbc19028a6373f9" candidate_uuid: "2b45210d2d4046b29e9e6114a6c96098" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "513a87b9102f402187ec511b32f1b3eb" is_pre_election: true
I20240501 14:09:37.025770 12759 raft_consensus.cc:2454] T 979ab8d0c9434e7dbfbc19028a6373f9 P 70fab6e24a8f4173bc4dbed6ddf9e8a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b45210d2d4046b29e9e6114a6c96098 in term 0.
I20240501 14:09:37.025869 13021 raft_consensus.cc:2454] T 979ab8d0c9434e7dbfbc19028a6373f9 P 513a87b9102f402187ec511b32f1b3eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b45210d2d4046b29e9e6114a6c96098 in term 0.
I20240501 14:09:37.026182 12825 leader_election.cc:302] T 979ab8d0c9434e7dbfbc19028a6373f9 P 2b45210d2d4046b29e9e6114a6c96098 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b45210d2d4046b29e9e6114a6c96098, 70fab6e24a8f4173bc4dbed6ddf9e8a6; no voters:
I20240501 14:09:37.026401 13091 raft_consensus.cc:2781] T 979ab8d0c9434e7dbfbc19028a6373f9 P 2b45210d2d4046b29e9e6114a6c96098 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:37.026472 13091 raft_consensus.cc:482] T 979ab8d0c9434e7dbfbc19028a6373f9 P 2b45210d2d4046b29e9e6114a6c96098 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:37.026518 13091 raft_consensus.cc:3037] T 979ab8d0c9434e7dbfbc19028a6373f9 P 2b45210d2d4046b29e9e6114a6c96098 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:37.027490 13091 raft_consensus.cc:504] T 979ab8d0c9434e7dbfbc19028a6373f9 P 2b45210d2d4046b29e9e6114a6c96098 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70fab6e24a8f4173bc4dbed6ddf9e8a6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 39579 } } peers { permanent_uuid: "2b45210d2d4046b29e9e6114a6c96098" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 34297 } } peers { permanent_uuid: "513a87b9102f402187ec511b32f1b3eb" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 33811 } }
I20240501 14:09:37.027827 13091 leader_election.cc:288] T 979ab8d0c9434e7dbfbc19028a6373f9 P 2b45210d2d4046b29e9e6114a6c96098 [CANDIDATE]: Term 1 election: Requested vote from peers 70fab6e24a8f4173bc4dbed6ddf9e8a6 (127.7.39.131:39579), 513a87b9102f402187ec511b32f1b3eb (127.7.39.133:33811)
I20240501 14:09:37.028141 12759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "979ab8d0c9434e7dbfbc19028a6373f9" candidate_uuid: "2b45210d2d4046b29e9e6114a6c96098" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70fab6e24a8f4173bc4dbed6ddf9e8a6"
I20240501 14:09:37.028192 13021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "979ab8d0c9434e7dbfbc19028a6373f9" candidate_uuid: "2b45210d2d4046b29e9e6114a6c96098" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "513a87b9102f402187ec511b32f1b3eb"
I20240501 14:09:37.028270 12759 raft_consensus.cc:3037] T 979ab8d0c9434e7dbfbc19028a6373f9 P 70fab6e24a8f4173bc4dbed6ddf9e8a6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:37.028329 13021 raft_consensus.cc:3037] T 979ab8d0c9434e7dbfbc19028a6373f9 P 513a87b9102f402187ec511b32f1b3eb [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:37.029181 12759 raft_consensus.cc:2454] T 979ab8d0c9434e7dbfbc19028a6373f9 P 70fab6e24a8f4173bc4dbed6ddf9e8a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b45210d2d4046b29e9e6114a6c96098 in term 1.
I20240501 14:09:37.029212 13021 raft_consensus.cc:2454] T 979ab8d0c9434e7dbfbc19028a6373f9 P 513a87b9102f402187ec511b32f1b3eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b45210d2d4046b29e9e6114a6c96098 in term 1.
I20240501 14:09:37.029486 12825 leader_election.cc:302] T 979ab8d0c9434e7dbfbc19028a6373f9 P 2b45210d2d4046b29e9e6114a6c96098 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b45210d2d4046b29e9e6114a6c96098, 70fab6e24a8f4173bc4dbed6ddf9e8a6; no voters:
I20240501 14:09:37.029671 13091 raft_consensus.cc:2781] T 979ab8d0c9434e7dbfbc19028a6373f9 P 2b45210d2d4046b29e9e6114a6c96098 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:37.029888 13091 raft_consensus.cc:686] T 979ab8d0c9434e7dbfbc19028a6373f9 P 2b45210d2d4046b29e9e6114a6c96098 [term 1 LEADER]: Becoming Leader. State: Replica: 2b45210d2d4046b29e9e6114a6c96098, State: Running, Role: LEADER
I20240501 14:09:37.030182 13091 consensus_queue.cc:237] T 979ab8d0c9434e7dbfbc19028a6373f9 P 2b45210d2d4046b29e9e6114a6c96098 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70fab6e24a8f4173bc4dbed6ddf9e8a6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 39579 } } peers { permanent_uuid: "2b45210d2d4046b29e9e6114a6c96098" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 34297 } } peers { permanent_uuid: "513a87b9102f402187ec511b32f1b3eb" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 33811 } }
I20240501 14:09:37.033233 12353 catalog_manager.cc:5484] T 979ab8d0c9434e7dbfbc19028a6373f9 P 2b45210d2d4046b29e9e6114a6c96098 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b45210d2d4046b29e9e6114a6c96098 (127.7.39.132). New cstate: current_term: 1 leader_uuid: "2b45210d2d4046b29e9e6114a6c96098" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70fab6e24a8f4173bc4dbed6ddf9e8a6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 39579 } } peers { permanent_uuid: "2b45210d2d4046b29e9e6114a6c96098" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 34297 } } peers { permanent_uuid: "513a87b9102f402187ec511b32f1b3eb" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 33811 } } }
I20240501 14:09:37.052225 7326 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 14:09:37.053830 7326 ts_itest-base.cc:233] Waiting for 0 tablets on tserver cf193f4215f74ad3959abcba3abf5ccc to finish bootstrapping
I20240501 14:09:37.059800 7326 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 04598e08a4654b7684ee8b47665ae4ac to finish bootstrapping
I20240501 14:09:37.064725 7326 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 70fab6e24a8f4173bc4dbed6ddf9e8a6 to finish bootstrapping
W20240501 14:09:37.065682 12806 tablet.cc:2369] T 979ab8d0c9434e7dbfbc19028a6373f9 P 70fab6e24a8f4173bc4dbed6ddf9e8a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:09:37.069041 7326 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 2b45210d2d4046b29e9e6114a6c96098 to finish bootstrapping
I20240501 14:09:37.073137 7326 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 513a87b9102f402187ec511b32f1b3eb to finish bootstrapping
W20240501 14:09:37.181856 13068 tablet.cc:2369] T 979ab8d0c9434e7dbfbc19028a6373f9 P 513a87b9102f402187ec511b32f1b3eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:09:37.210969 12543 heartbeater.cc:503] Master 127.7.39.190:34995 was elected leader, sending a full tablet report...
W20240501 14:09:37.242337 12937 tablet.cc:2369] T 979ab8d0c9434e7dbfbc19028a6373f9 P 2b45210d2d4046b29e9e6114a6c96098: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:09:37.257035 13001 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:37.257057 12608 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:37.257051 12870 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:37.258546 12739 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:37.265173 12477 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:37.347690 12608 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:37.349215 12870 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:37.350545 12477 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:37.351809 12739 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:37.352232 13001 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:37.397471 12674 heartbeater.cc:503] Master 127.7.39.190:34995 was elected leader, sending a full tablet report...
I20240501 14:09:37.416889 13091 consensus_queue.cc:1035] T 979ab8d0c9434e7dbfbc19028a6373f9 P 2b45210d2d4046b29e9e6114a6c96098 [LEADER]: Connected to new peer: Peer: permanent_uuid: "513a87b9102f402187ec511b32f1b3eb" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 33811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:37.422101 13094 consensus_queue.cc:1035] T 979ab8d0c9434e7dbfbc19028a6373f9 P 2b45210d2d4046b29e9e6114a6c96098 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70fab6e24a8f4173bc4dbed6ddf9e8a6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 39579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:37.644467 12870 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:37.646941 13001 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:37.647030 12608 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:37.661319 12477 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:37.662276 12739 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:37.758929 12608 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:37.759006 12870 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:37.759900 13001 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:37.761816 12739 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:37.766454 12477 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:37.865319 12608 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:37.865389 12870 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:37.865731 12739 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:37.867126 13001 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:37.868494 12477 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:37.907161 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 12415
I20240501 14:09:37.913024 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 12546
I20240501 14:09:37.918951 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 12677
I20240501 14:09:37.924925 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 12808
I20240501 14:09:37.930577 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 12939
I20240501 14:09:37.936487 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 12324
2024-05-01T14:09:37Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/0 (2182 ms)
[----------] 1 test from RebalanceStartCriteriaTest (2182 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/2
I20240501 14:09:37.959529 7326 test_util.cc:268] Using random seed: 158559844
I20240501 14:09:37.960278 7326 ts_itest-base.cc:112] Starting cluster with:
I20240501 14:09:37.960318 7326 ts_itest-base.cc:113] --------------
I20240501 14:09:37.960330 7326 ts_itest-base.cc:114] 7 tablet servers
I20240501 14:09:37.960340 7326 ts_itest-base.cc:115] 2 replicas per TS
I20240501 14:09:37.960350 7326 ts_itest-base.cc:116] --------------
2024-05-01T14:09:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T14:09:37Z Disabled control of system clock
I20240501 14:09:37.968645 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.39.190:41789
--webserver_interface=127.7.39.190
--webserver_port=0
--builtin_ntp_servers=127.7.39.148:33673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.39.190:41789
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240501 14:09:38.072867 13273 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240501 14:09:38.073123 13273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:38.073216 13273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:38.073284 13273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:38.076534 13273 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:09:38.076607 13273 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 14:09:38.076632 13273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:38.076653 13273 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 14:09:38.076675 13273 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 14:09:38.080346 13273 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:33673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.7.39.190:41789
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.39.190:41789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.7.39.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.13273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:38.081228 13273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:38.082289 13273 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:38.090167 13278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:38.091439 13281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:38.092515 13279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:38.093636 13273 server_base.cc:1034] running on GCE node
I20240501 14:09:38.094172 13273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:38.095170 13273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:38.096367 13273 hybrid_clock.cc:648] HybridClock initialized: now 1714572578096347 us; error 40 us; skew 500 ppm
I20240501 14:09:38.098379 13273 webserver.cc:460] Webserver started at http://127.7.39.190:42417/ using document root <none> and password file <none>
I20240501 14:09:38.098918 13273 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:38.098981 13273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:38.099195 13273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:38.100836 13273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "f86e915e5bb5479aa7f3e095cf9884ba"
format_stamp: "Formatted at 2024-05-01 14:09:38 on dist-test-slave-rnfw"
I20240501 14:09:38.101300 13273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "f86e915e5bb5479aa7f3e095cf9884ba"
format_stamp: "Formatted at 2024-05-01 14:09:38 on dist-test-slave-rnfw"
I20240501 14:09:38.104490 13273 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20240501 14:09:38.106530 13287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:38.107574 13273 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:38.107690 13273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/wal
uuid: "f86e915e5bb5479aa7f3e095cf9884ba"
format_stamp: "Formatted at 2024-05-01 14:09:38 on dist-test-slave-rnfw"
I20240501 14:09:38.107779 13273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:38.123401 13273 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:38.124100 13273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:38.124315 13273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:38.131109 13339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.190:41789 every 8 connection(s)
I20240501 14:09:38.131116 13273 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.190:41789
I20240501 14:09:38.132180 13273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 14:09:38.133939 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 13273
I20240501 14:09:38.134060 7326 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 14:09:38.134912 13340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.141263 13340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f86e915e5bb5479aa7f3e095cf9884ba: Bootstrap starting.
I20240501 14:09:38.143330 13340 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f86e915e5bb5479aa7f3e095cf9884ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.143970 13340 log.cc:824] T 00000000000000000000000000000000 P f86e915e5bb5479aa7f3e095cf9884ba: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:38.145505 13340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f86e915e5bb5479aa7f3e095cf9884ba: No bootstrap required, opened a new log
I20240501 14:09:38.147939 13340 raft_consensus.cc:348] T 00000000000000000000000000000000 P f86e915e5bb5479aa7f3e095cf9884ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f86e915e5bb5479aa7f3e095cf9884ba" member_type: VOTER last_known_addr { host: "127.7.39.190" port: 41789 } }
I20240501 14:09:38.148118 13340 raft_consensus.cc:374] T 00000000000000000000000000000000 P f86e915e5bb5479aa7f3e095cf9884ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.148186 13340 raft_consensus.cc:729] T 00000000000000000000000000000000 P f86e915e5bb5479aa7f3e095cf9884ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f86e915e5bb5479aa7f3e095cf9884ba, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.148551 13340 consensus_queue.cc:260] T 00000000000000000000000000000000 P f86e915e5bb5479aa7f3e095cf9884ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f86e915e5bb5479aa7f3e095cf9884ba" member_type: VOTER last_known_addr { host: "127.7.39.190" port: 41789 } }
I20240501 14:09:38.148669 13340 raft_consensus.cc:388] T 00000000000000000000000000000000 P f86e915e5bb5479aa7f3e095cf9884ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:09:38.148732 13340 raft_consensus.cc:482] T 00000000000000000000000000000000 P f86e915e5bb5479aa7f3e095cf9884ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:09:38.148847 13340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f86e915e5bb5479aa7f3e095cf9884ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.149566 13340 raft_consensus.cc:504] T 00000000000000000000000000000000 P f86e915e5bb5479aa7f3e095cf9884ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f86e915e5bb5479aa7f3e095cf9884ba" member_type: VOTER last_known_addr { host: "127.7.39.190" port: 41789 } }
I20240501 14:09:38.149857 13340 leader_election.cc:302] T 00000000000000000000000000000000 P f86e915e5bb5479aa7f3e095cf9884ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f86e915e5bb5479aa7f3e095cf9884ba; no voters:
I20240501 14:09:38.150108 13340 leader_election.cc:288] T 00000000000000000000000000000000 P f86e915e5bb5479aa7f3e095cf9884ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:09:38.150480 13345 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f86e915e5bb5479aa7f3e095cf9884ba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:38.150696 13345 raft_consensus.cc:686] T 00000000000000000000000000000000 P f86e915e5bb5479aa7f3e095cf9884ba [term 1 LEADER]: Becoming Leader. State: Replica: f86e915e5bb5479aa7f3e095cf9884ba, State: Running, Role: LEADER
I20240501 14:09:38.150974 13345 consensus_queue.cc:237] T 00000000000000000000000000000000 P f86e915e5bb5479aa7f3e095cf9884ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f86e915e5bb5479aa7f3e095cf9884ba" member_type: VOTER last_known_addr { host: "127.7.39.190" port: 41789 } }
I20240501 14:09:38.151154 13340 sys_catalog.cc:564] T 00000000000000000000000000000000 P f86e915e5bb5479aa7f3e095cf9884ba [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:09:38.152601 13346 sys_catalog.cc:455] T 00000000000000000000000000000000 P f86e915e5bb5479aa7f3e095cf9884ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f86e915e5bb5479aa7f3e095cf9884ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f86e915e5bb5479aa7f3e095cf9884ba" member_type: VOTER last_known_addr { host: "127.7.39.190" port: 41789 } } }
I20240501 14:09:38.152616 13347 sys_catalog.cc:455] T 00000000000000000000000000000000 P f86e915e5bb5479aa7f3e095cf9884ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader f86e915e5bb5479aa7f3e095cf9884ba. Latest consensus state: current_term: 1 leader_uuid: "f86e915e5bb5479aa7f3e095cf9884ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f86e915e5bb5479aa7f3e095cf9884ba" member_type: VOTER last_known_addr { host: "127.7.39.190" port: 41789 } } }
I20240501 14:09:38.152760 13347 sys_catalog.cc:458] T 00000000000000000000000000000000 P f86e915e5bb5479aa7f3e095cf9884ba [sys.catalog]: This master's current role is: LEADER
I20240501 14:09:38.152760 13346 sys_catalog.cc:458] T 00000000000000000000000000000000 P f86e915e5bb5479aa7f3e095cf9884ba [sys.catalog]: This master's current role is: LEADER
W20240501 14:09:38.156009 13360 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f86e915e5bb5479aa7f3e095cf9884ba: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 14:09:38.156092 13360 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 14:09:38.156209 13358 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:09:38.156960 13358 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:09:38.161705 13358 catalog_manager.cc:1347] Generated new cluster ID: 3e98f76650114874b71b5f20a4ca250d
I20240501 14:09:38.161779 13358 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:09:38.167778 13358 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 14:09:38.168874 13358 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:09:38.176859 13358 catalog_manager.cc:5857] T 00000000000000000000000000000000 P f86e915e5bb5479aa7f3e095cf9884ba: Generated new TSK 0
I20240501 14:09:38.177538 13358 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:09:38.184356 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.129:0
--local_ip_for_outbound_sockets=127.7.39.129
--webserver_interface=127.7.39.129
--webserver_port=0
--tserver_master_addrs=127.7.39.190:41789
--builtin_ntp_servers=127.7.39.148:33673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 14:09:38.294131 13364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:38.294363 13364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:38.294436 13364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:38.297435 13364 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:09:38.297518 13364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:38.297616 13364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.129
I20240501 14:09:38.301244 13364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:33673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.7.39.129
--webserver_port=0
--tserver_master_addrs=127.7.39.190:41789
--never_fsync=true
--heap_profile_path=/tmp/kudu.13364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.129
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:38.302129 13364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:38.303287 13364 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:38.312775 13369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:38.314042 13370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:38.315128 13372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:38.316373 13364 server_base.cc:1034] running on GCE node
I20240501 14:09:38.316828 13364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:38.317413 13364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:38.318594 13364 hybrid_clock.cc:648] HybridClock initialized: now 1714572578318574 us; error 35 us; skew 500 ppm
I20240501 14:09:38.320780 13364 webserver.cc:460] Webserver started at http://127.7.39.129:42305/ using document root <none> and password file <none>
I20240501 14:09:38.321350 13364 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:38.321414 13364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:38.321604 13364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:38.323354 13364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "50d514f034204fc38331aa0a4fd74a1b"
format_stamp: "Formatted at 2024-05-01 14:09:38 on dist-test-slave-rnfw"
I20240501 14:09:38.323851 13364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "50d514f034204fc38331aa0a4fd74a1b"
format_stamp: "Formatted at 2024-05-01 14:09:38 on dist-test-slave-rnfw"
I20240501 14:09:38.327232 13364 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20240501 14:09:38.329416 13378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:38.330397 13364 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:38.330519 13364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "50d514f034204fc38331aa0a4fd74a1b"
format_stamp: "Formatted at 2024-05-01 14:09:38 on dist-test-slave-rnfw"
I20240501 14:09:38.330611 13364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:38.353446 13364 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:38.354139 13364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:38.354316 13364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:38.354871 13364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:38.355800 13364 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:38.355849 13364 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:38.355891 13364 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:38.355926 13364 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:38.364080 13364 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.129:37933
I20240501 14:09:38.364140 13491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.129:37933 every 8 connection(s)
I20240501 14:09:38.365036 13364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 14:09:38.370966 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 13364
I20240501 14:09:38.371122 7326 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 14:09:38.373276 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.130:0
--local_ip_for_outbound_sockets=127.7.39.130
--webserver_interface=127.7.39.130
--webserver_port=0
--tserver_master_addrs=127.7.39.190:41789
--builtin_ntp_servers=127.7.39.148:33673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:09:38.375190 13492 heartbeater.cc:347] Connected to a master server at 127.7.39.190:41789
I20240501 14:09:38.375453 13492 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:38.375996 13492 heartbeater.cc:511] Master 127.7.39.190:41789 requested a full tablet report, sending...
I20240501 14:09:38.377369 13304 ts_manager.cc:194] Registered new tserver with Master: 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129:37933)
I20240501 14:09:38.378530 13304 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.129:43687
W20240501 14:09:38.481437 13495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:38.481660 13495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:38.481703 13495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:38.484630 13495 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:09:38.484699 13495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:38.484784 13495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.130
I20240501 14:09:38.488446 13495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:33673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.7.39.130
--webserver_port=0
--tserver_master_addrs=127.7.39.190:41789
--never_fsync=true
--heap_profile_path=/tmp/kudu.13495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.130
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:38.489475 13495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:38.490526 13495 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:38.499635 13500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:38.500660 13503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:38.501984 13501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:38.503116 13495 server_base.cc:1034] running on GCE node
I20240501 14:09:38.503511 13495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:38.504099 13495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:38.505264 13495 hybrid_clock.cc:648] HybridClock initialized: now 1714572578505242 us; error 36 us; skew 500 ppm
I20240501 14:09:38.507143 13495 webserver.cc:460] Webserver started at http://127.7.39.130:39177/ using document root <none> and password file <none>
I20240501 14:09:38.507692 13495 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:38.507748 13495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:38.507923 13495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:38.509500 13495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "43736346b1ff48c78ac919c80ba58654"
format_stamp: "Formatted at 2024-05-01 14:09:38 on dist-test-slave-rnfw"
I20240501 14:09:38.509953 13495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "43736346b1ff48c78ac919c80ba58654"
format_stamp: "Formatted at 2024-05-01 14:09:38 on dist-test-slave-rnfw"
I20240501 14:09:38.513298 13495 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 14:09:38.515450 13509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:38.516420 13495 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:09:38.516520 13495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "43736346b1ff48c78ac919c80ba58654"
format_stamp: "Formatted at 2024-05-01 14:09:38 on dist-test-slave-rnfw"
I20240501 14:09:38.516645 13495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:38.530967 13495 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:38.531620 13495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:38.531776 13495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:38.532302 13495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:38.533178 13495 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:38.533226 13495 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:38.533296 13495 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:38.533331 13495 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:38.541393 13495 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.130:43311
I20240501 14:09:38.541438 13622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.130:43311 every 8 connection(s)
I20240501 14:09:38.542286 13495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 14:09:38.550524 13623 heartbeater.cc:347] Connected to a master server at 127.7.39.190:41789
I20240501 14:09:38.550673 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 13495
I20240501 14:09:38.550782 7326 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 14:09:38.550802 13623 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:38.551369 13623 heartbeater.cc:511] Master 127.7.39.190:41789 requested a full tablet report, sending...
I20240501 14:09:38.552340 13303 ts_manager.cc:194] Registered new tserver with Master: 43736346b1ff48c78ac919c80ba58654 (127.7.39.130:43311)
I20240501 14:09:38.552994 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.131:0
--local_ip_for_outbound_sockets=127.7.39.131
--webserver_interface=127.7.39.131
--webserver_port=0
--tserver_master_addrs=127.7.39.190:41789
--builtin_ntp_servers=127.7.39.148:33673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:09:38.553084 13303 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.130:37827
W20240501 14:09:38.663482 13626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:38.663697 13626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:38.663744 13626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:38.666699 13626 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:09:38.666765 13626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:38.666862 13626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.131
I20240501 14:09:38.670346 13626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:33673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.7.39.131
--webserver_port=0
--tserver_master_addrs=127.7.39.190:41789
--never_fsync=true
--heap_profile_path=/tmp/kudu.13626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.131
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:38.671249 13626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:38.672335 13626 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:38.681123 13634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:38.682410 13632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:38.683447 13631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:38.684777 13626 server_base.cc:1034] running on GCE node
I20240501 14:09:38.685142 13626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:38.685683 13626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:38.686857 13626 hybrid_clock.cc:648] HybridClock initialized: now 1714572578686818 us; error 51 us; skew 500 ppm
I20240501 14:09:38.688776 13626 webserver.cc:460] Webserver started at http://127.7.39.131:43033/ using document root <none> and password file <none>
I20240501 14:09:38.689301 13626 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:38.689364 13626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:38.689543 13626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:38.691251 13626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "a278448771e14471ab4df1e6649485bd"
format_stamp: "Formatted at 2024-05-01 14:09:38 on dist-test-slave-rnfw"
I20240501 14:09:38.691738 13626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "a278448771e14471ab4df1e6649485bd"
format_stamp: "Formatted at 2024-05-01 14:09:38 on dist-test-slave-rnfw"
I20240501 14:09:38.694941 13626 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:09:38.697211 13640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:38.698175 13626 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:38.698289 13626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "a278448771e14471ab4df1e6649485bd"
format_stamp: "Formatted at 2024-05-01 14:09:38 on dist-test-slave-rnfw"
I20240501 14:09:38.698370 13626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:38.709796 13626 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:38.710407 13626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:38.710562 13626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:38.711119 13626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:38.711989 13626 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:38.712081 13626 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:38.712134 13626 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:38.712149 13626 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:38.720788 13626 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.131:46461
I20240501 14:09:38.720815 13753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.131:46461 every 8 connection(s)
I20240501 14:09:38.721727 13626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 14:09:38.729231 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 13626
I20240501 14:09:38.729334 7326 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 14:09:38.729776 13754 heartbeater.cc:347] Connected to a master server at 127.7.39.190:41789
I20240501 14:09:38.729967 13754 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:38.730440 13754 heartbeater.cc:511] Master 127.7.39.190:41789 requested a full tablet report, sending...
I20240501 14:09:38.731387 13303 ts_manager.cc:194] Registered new tserver with Master: a278448771e14471ab4df1e6649485bd (127.7.39.131:46461)
I20240501 14:09:38.731657 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.132:0
--local_ip_for_outbound_sockets=127.7.39.132
--webserver_interface=127.7.39.132
--webserver_port=0
--tserver_master_addrs=127.7.39.190:41789
--builtin_ntp_servers=127.7.39.148:33673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:09:38.732071 13303 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.131:37379
W20240501 14:09:38.837236 13757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:38.837471 13757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:38.837548 13757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:38.840528 13757 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:09:38.840597 13757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:38.840692 13757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.132
I20240501 14:09:38.844308 13757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:33673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.7.39.132
--webserver_port=0
--tserver_master_addrs=127.7.39.190:41789
--never_fsync=true
--heap_profile_path=/tmp/kudu.13757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.132
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:38.845186 13757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:38.846298 13757 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:38.855362 13765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:38.856295 13762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:38.857666 13763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:38.858781 13757 server_base.cc:1034] running on GCE node
I20240501 14:09:38.859192 13757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:38.859771 13757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:38.860927 13757 hybrid_clock.cc:648] HybridClock initialized: now 1714572578860908 us; error 34 us; skew 500 ppm
I20240501 14:09:38.862819 13757 webserver.cc:460] Webserver started at http://127.7.39.132:40497/ using document root <none> and password file <none>
I20240501 14:09:38.863415 13757 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:38.863497 13757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:38.863711 13757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:38.865350 13757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "2496e4af45ba458eacaacc9d25dcf01a"
format_stamp: "Formatted at 2024-05-01 14:09:38 on dist-test-slave-rnfw"
I20240501 14:09:38.865854 13757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "2496e4af45ba458eacaacc9d25dcf01a"
format_stamp: "Formatted at 2024-05-01 14:09:38 on dist-test-slave-rnfw"
I20240501 14:09:38.869195 13757 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 14:09:38.871361 13771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:38.872298 13757 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 14:09:38.872400 13757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "2496e4af45ba458eacaacc9d25dcf01a"
format_stamp: "Formatted at 2024-05-01 14:09:38 on dist-test-slave-rnfw"
I20240501 14:09:38.872506 13757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:38.896224 13757 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:38.896909 13757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:38.897078 13757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:38.897647 13757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:38.898564 13757 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:38.898614 13757 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:38.898675 13757 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:38.898712 13757 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:38.907477 13757 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.132:43977
I20240501 14:09:38.907511 13884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.132:43977 every 8 connection(s)
I20240501 14:09:38.908423 13757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 14:09:38.916819 13885 heartbeater.cc:347] Connected to a master server at 127.7.39.190:41789
I20240501 14:09:38.917035 13885 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:38.917457 13885 heartbeater.cc:511] Master 127.7.39.190:41789 requested a full tablet report, sending...
I20240501 14:09:38.918103 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 13757
I20240501 14:09:38.918205 7326 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 14:09:38.918416 13303 ts_manager.cc:194] Registered new tserver with Master: 2496e4af45ba458eacaacc9d25dcf01a (127.7.39.132:43977)
I20240501 14:09:38.919095 13303 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.132:43533
I20240501 14:09:38.920316 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.133:0
--local_ip_for_outbound_sockets=127.7.39.133
--webserver_interface=127.7.39.133
--webserver_port=0
--tserver_master_addrs=127.7.39.190:41789
--builtin_ntp_servers=127.7.39.148:33673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 14:09:39.027061 13888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:39.027328 13888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:39.027376 13888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:39.030390 13888 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:09:39.030454 13888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:39.030541 13888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.133
I20240501 14:09:39.033967 13888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:33673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.7.39.133
--webserver_port=0
--tserver_master_addrs=127.7.39.190:41789
--never_fsync=true
--heap_profile_path=/tmp/kudu.13888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.133
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:39.034864 13888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:39.036060 13888 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:39.045014 13896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:39.045923 13893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:39.047351 13894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:39.048462 13888 server_base.cc:1034] running on GCE node
I20240501 14:09:39.048842 13888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:39.049434 13888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:39.050614 13888 hybrid_clock.cc:648] HybridClock initialized: now 1714572579050595 us; error 38 us; skew 500 ppm
I20240501 14:09:39.052775 13888 webserver.cc:460] Webserver started at http://127.7.39.133:42977/ using document root <none> and password file <none>
I20240501 14:09:39.053443 13888 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:39.053519 13888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:39.053725 13888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:39.055560 13888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "4015e9df09424332bc3d7a9bad781641"
format_stamp: "Formatted at 2024-05-01 14:09:39 on dist-test-slave-rnfw"
I20240501 14:09:39.056048 13888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "4015e9df09424332bc3d7a9bad781641"
format_stamp: "Formatted at 2024-05-01 14:09:39 on dist-test-slave-rnfw"
I20240501 14:09:39.059348 13888 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:09:39.062359 13902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:39.063779 13888 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:09:39.063903 13888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "4015e9df09424332bc3d7a9bad781641"
format_stamp: "Formatted at 2024-05-01 14:09:39 on dist-test-slave-rnfw"
I20240501 14:09:39.064004 13888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:39.086126 13888 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:39.086787 13888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:39.086954 13888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:39.087553 13888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:39.088435 13888 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:39.088502 13888 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:39.088562 13888 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:39.088596 13888 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:39.096940 13888 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.133:42905
I20240501 14:09:39.096972 14015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.133:42905 every 8 connection(s)
I20240501 14:09:39.097863 13888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 14:09:39.106112 14016 heartbeater.cc:347] Connected to a master server at 127.7.39.190:41789
I20240501 14:09:39.106330 14016 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:39.106534 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 13888
I20240501 14:09:39.106639 7326 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 14:09:39.106762 14016 heartbeater.cc:511] Master 127.7.39.190:41789 requested a full tablet report, sending...
I20240501 14:09:39.107774 13303 ts_manager.cc:194] Registered new tserver with Master: 4015e9df09424332bc3d7a9bad781641 (127.7.39.133:42905)
I20240501 14:09:39.108316 13303 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.133:48699
I20240501 14:09:39.108696 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.134:0
--local_ip_for_outbound_sockets=127.7.39.134
--webserver_interface=127.7.39.134
--webserver_port=0
--tserver_master_addrs=127.7.39.190:41789
--builtin_ntp_servers=127.7.39.148:33673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 14:09:39.214896 14019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:39.215157 14019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:39.215206 14019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:39.218235 14019 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:09:39.218297 14019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:39.218364 14019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.134
I20240501 14:09:39.222059 14019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:33673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.7.39.134
--webserver_port=0
--tserver_master_addrs=127.7.39.190:41789
--never_fsync=true
--heap_profile_path=/tmp/kudu.14019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.134
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:39.222987 14019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:39.224175 14019 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:39.233019 14025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:39.234429 14027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:39.235468 14024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:39.236728 14019 server_base.cc:1034] running on GCE node
I20240501 14:09:39.237088 14019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:39.237668 14019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:39.238819 14019 hybrid_clock.cc:648] HybridClock initialized: now 1714572579238807 us; error 28 us; skew 500 ppm
I20240501 14:09:39.240792 14019 webserver.cc:460] Webserver started at http://127.7.39.134:36143/ using document root <none> and password file <none>
I20240501 14:09:39.241315 14019 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:39.241372 14019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:39.241575 14019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:39.243489 14019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "e26df472efe547f8bc5951dce352471b"
format_stamp: "Formatted at 2024-05-01 14:09:39 on dist-test-slave-rnfw"
I20240501 14:09:39.244428 14019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "e26df472efe547f8bc5951dce352471b"
format_stamp: "Formatted at 2024-05-01 14:09:39 on dist-test-slave-rnfw"
I20240501 14:09:39.247895 14019 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:09:39.250183 14033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:39.251346 14019 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:09:39.251477 14019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "e26df472efe547f8bc5951dce352471b"
format_stamp: "Formatted at 2024-05-01 14:09:39 on dist-test-slave-rnfw"
I20240501 14:09:39.251566 14019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:39.269579 14019 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:39.270263 14019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:39.270560 14019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:39.271256 14019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:39.272199 14019 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:39.272251 14019 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:39.272317 14019 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:39.272344 14019 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:39.281287 14019 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.134:33035
I20240501 14:09:39.281318 14146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.134:33035 every 8 connection(s)
I20240501 14:09:39.282194 14019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 14:09:39.284873 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 14019
I20240501 14:09:39.284998 7326 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 14:09:39.287431 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.135:0
--local_ip_for_outbound_sockets=127.7.39.135
--webserver_interface=127.7.39.135
--webserver_port=0
--tserver_master_addrs=127.7.39.190:41789
--builtin_ntp_servers=127.7.39.148:33673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:09:39.291863 14147 heartbeater.cc:347] Connected to a master server at 127.7.39.190:41789
I20240501 14:09:39.292073 14147 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:39.292515 14147 heartbeater.cc:511] Master 127.7.39.190:41789 requested a full tablet report, sending...
I20240501 14:09:39.293444 13303 ts_manager.cc:194] Registered new tserver with Master: e26df472efe547f8bc5951dce352471b (127.7.39.134:33035)
I20240501 14:09:39.293989 13303 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.134:49897
I20240501 14:09:39.380719 13492 heartbeater.cc:503] Master 127.7.39.190:41789 was elected leader, sending a full tablet report...
W20240501 14:09:39.393575 14150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:39.393783 14150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:39.393826 14150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:39.396816 14150 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:09:39.396893 14150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:39.396965 14150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.135
I20240501 14:09:39.400528 14150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:33673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.7.39.135
--webserver_port=0
--tserver_master_addrs=127.7.39.190:41789
--never_fsync=true
--heap_profile_path=/tmp/kudu.14150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.135
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:39.401417 14150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:39.402590 14150 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:39.411331 14158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:39.412822 14155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:39.413791 14156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:39.415107 14150 server_base.cc:1034] running on GCE node
I20240501 14:09:39.415498 14150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:39.416049 14150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:39.417212 14150 hybrid_clock.cc:648] HybridClock initialized: now 1714572579417200 us; error 33 us; skew 500 ppm
I20240501 14:09:39.419221 14150 webserver.cc:460] Webserver started at http://127.7.39.135:35767/ using document root <none> and password file <none>
I20240501 14:09:39.419785 14150 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:39.419847 14150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:39.420056 14150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:39.421723 14150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "39bde7c058154f8aa40085db77f8c6e5"
format_stamp: "Formatted at 2024-05-01 14:09:39 on dist-test-slave-rnfw"
I20240501 14:09:39.422169 14150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "39bde7c058154f8aa40085db77f8c6e5"
format_stamp: "Formatted at 2024-05-01 14:09:39 on dist-test-slave-rnfw"
I20240501 14:09:39.425360 14150 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 14:09:39.427459 14164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:39.428426 14150 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:39.428529 14150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "39bde7c058154f8aa40085db77f8c6e5"
format_stamp: "Formatted at 2024-05-01 14:09:39 on dist-test-slave-rnfw"
I20240501 14:09:39.428606 14150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:39.441263 14150 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:39.441916 14150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:39.442062 14150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:39.442559 14150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:39.443442 14150 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:39.443495 14150 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:39.443552 14150 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:39.443575 14150 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:39.451498 14150 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.135:37645
I20240501 14:09:39.451534 14277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.135:37645 every 8 connection(s)
I20240501 14:09:39.452553 14150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 14:09:39.453464 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 14150
I20240501 14:09:39.453555 7326 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 14:09:39.461354 14278 heartbeater.cc:347] Connected to a master server at 127.7.39.190:41789
I20240501 14:09:39.461583 14278 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:39.462008 14278 heartbeater.cc:511] Master 127.7.39.190:41789 requested a full tablet report, sending...
I20240501 14:09:39.462865 13303 ts_manager.cc:194] Registered new tserver with Master: 39bde7c058154f8aa40085db77f8c6e5 (127.7.39.135:37645)
I20240501 14:09:39.463449 13303 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.135:58295
I20240501 14:09:39.467973 7326 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240501 14:09:39.476943 13303 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43662:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 14:09:39.493945 13557 tablet_service.cc:1467] Processing CreateTablet for tablet b0eb8ac1778a40a0a0e3c6b113ce5741 (DEFAULT_TABLE table=TestTable [id=59a0cdfa526143ab84978a90fb8bc85b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:39.493978 14081 tablet_service.cc:1467] Processing CreateTablet for tablet b0eb8ac1778a40a0a0e3c6b113ce5741 (DEFAULT_TABLE table=TestTable [id=59a0cdfa526143ab84978a90fb8bc85b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:39.494843 13557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0eb8ac1778a40a0a0e3c6b113ce5741. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.494843 14081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0eb8ac1778a40a0a0e3c6b113ce5741. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.499780 14294 tablet_bootstrap.cc:492] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654: Bootstrap starting.
I20240501 14:09:39.499837 14295 tablet_bootstrap.cc:492] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b: Bootstrap starting.
I20240501 14:09:39.501936 14295 tablet_bootstrap.cc:654] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:39.501936 14294 tablet_bootstrap.cc:654] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:39.502583 14295 log.cc:824] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:39.502719 14294 log.cc:824] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:39.504104 14295 tablet_bootstrap.cc:492] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b: No bootstrap required, opened a new log
I20240501 14:09:39.504225 14294 tablet_bootstrap.cc:492] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654: No bootstrap required, opened a new log
I20240501 14:09:39.504244 14295 ts_tablet_manager.cc:1367] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 14:09:39.504385 14294 ts_tablet_manager.cc:1367] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 14:09:39.506604 14294 raft_consensus.cc:348] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } }
I20240501 14:09:39.506805 14294 raft_consensus.cc:374] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:39.506888 14294 raft_consensus.cc:729] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43736346b1ff48c78ac919c80ba58654, State: Initialized, Role: FOLLOWER
I20240501 14:09:39.506877 14295 raft_consensus.cc:348] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } }
I20240501 14:09:39.507091 14295 raft_consensus.cc:374] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:39.507158 14295 raft_consensus.cc:729] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e26df472efe547f8bc5951dce352471b, State: Initialized, Role: FOLLOWER
I20240501 14:09:39.507359 14294 consensus_queue.cc:260] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } }
I20240501 14:09:39.507526 14295 consensus_queue.cc:260] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } }
I20240501 14:09:39.508013 13623 heartbeater.cc:503] Master 127.7.39.190:41789 was elected leader, sending a full tablet report...
I20240501 14:09:39.508280 14294 ts_tablet_manager.cc:1398] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 14:09:39.508306 14295 ts_tablet_manager.cc:1398] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b: Time spent starting tablet: real 0.004s user 0.000s sys 0.005s
I20240501 14:09:39.508719 14147 heartbeater.cc:503] Master 127.7.39.190:41789 was elected leader, sending a full tablet report...
W20240501 14:09:39.533100 14148 tablet.cc:2369] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:09:39.543768 13624 tablet.cc:2369] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:09:39.734300 13754 heartbeater.cc:503] Master 127.7.39.190:41789 was elected leader, sending a full tablet report...
I20240501 14:09:39.788424 14299 raft_consensus.cc:482] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:39.788630 14299 raft_consensus.cc:504] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } }
I20240501 14:09:39.789677 14299 leader_election.cc:288] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43736346b1ff48c78ac919c80ba58654 (127.7.39.130:43311)
I20240501 14:09:39.792886 13577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0eb8ac1778a40a0a0e3c6b113ce5741" candidate_uuid: "e26df472efe547f8bc5951dce352471b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43736346b1ff48c78ac919c80ba58654" is_pre_election: true
I20240501 14:09:39.793145 13577 raft_consensus.cc:2454] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e26df472efe547f8bc5951dce352471b in term 0.
I20240501 14:09:39.793546 14037 leader_election.cc:302] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 43736346b1ff48c78ac919c80ba58654, e26df472efe547f8bc5951dce352471b; no voters:
I20240501 14:09:39.793767 14299 raft_consensus.cc:2781] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:39.793849 14299 raft_consensus.cc:482] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:39.793901 14299 raft_consensus.cc:3037] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:39.794821 14299 raft_consensus.cc:504] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } }
I20240501 14:09:39.795138 14299 leader_election.cc:288] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b [CANDIDATE]: Term 1 election: Requested vote from peers 43736346b1ff48c78ac919c80ba58654 (127.7.39.130:43311)
I20240501 14:09:39.795498 13577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0eb8ac1778a40a0a0e3c6b113ce5741" candidate_uuid: "e26df472efe547f8bc5951dce352471b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43736346b1ff48c78ac919c80ba58654"
I20240501 14:09:39.795600 13577 raft_consensus.cc:3037] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:39.796427 13577 raft_consensus.cc:2454] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e26df472efe547f8bc5951dce352471b in term 1.
I20240501 14:09:39.796708 14037 leader_election.cc:302] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 43736346b1ff48c78ac919c80ba58654, e26df472efe547f8bc5951dce352471b; no voters:
I20240501 14:09:39.796857 14299 raft_consensus.cc:2781] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:39.797032 14299 raft_consensus.cc:686] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b [term 1 LEADER]: Becoming Leader. State: Replica: e26df472efe547f8bc5951dce352471b, State: Running, Role: LEADER
I20240501 14:09:39.797299 14299 consensus_queue.cc:237] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } }
I20240501 14:09:39.799819 13303 catalog_manager.cc:5484] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b reported cstate change: term changed from 0 to 1, leader changed from <none> to e26df472efe547f8bc5951dce352471b (127.7.39.134). New cstate: current_term: 1 leader_uuid: "e26df472efe547f8bc5951dce352471b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } }
I20240501 14:09:39.849697 7326 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240501 14:09:39.851240 7326 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 50d514f034204fc38331aa0a4fd74a1b to finish bootstrapping
I20240501 14:09:39.857784 7326 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 43736346b1ff48c78ac919c80ba58654 to finish bootstrapping
I20240501 14:09:39.862715 7326 ts_itest-base.cc:233] Waiting for 0 tablets on tserver a278448771e14471ab4df1e6649485bd to finish bootstrapping
I20240501 14:09:39.868054 7326 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 2496e4af45ba458eacaacc9d25dcf01a to finish bootstrapping
I20240501 14:09:39.873555 7326 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 4015e9df09424332bc3d7a9bad781641 to finish bootstrapping
I20240501 14:09:39.879215 7326 ts_itest-base.cc:233] Waiting for 1 tablets on tserver e26df472efe547f8bc5951dce352471b to finish bootstrapping
I20240501 14:09:39.883793 7326 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 39bde7c058154f8aa40085db77f8c6e5 to finish bootstrapping
I20240501 14:09:39.889235 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 13757
I20240501 14:09:39.894802 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 13888
I20240501 14:09:39.900468 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 14019
I20240501 14:09:39.906199 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 14150
I20240501 14:09:41.511327 14308 raft_consensus.cc:482] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:41.511485 14308 raft_consensus.cc:504] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } }
I20240501 14:09:41.512486 14308 leader_election.cc:288] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e26df472efe547f8bc5951dce352471b (127.7.39.134:33035)
W20240501 14:09:41.513273 13511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:33035: connect: Connection refused (error 111)
W20240501 14:09:41.514117 13511 leader_election.cc:334] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e26df472efe547f8bc5951dce352471b (127.7.39.134:33035): Network error: Client connection negotiation failed: client connection to 127.7.39.134:33035: connect: Connection refused (error 111)
I20240501 14:09:41.514210 13511 leader_election.cc:302] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 43736346b1ff48c78ac919c80ba58654; no voters: e26df472efe547f8bc5951dce352471b
I20240501 14:09:41.514433 14308 raft_consensus.cc:2726] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:09:43.038193 14311 raft_consensus.cc:482] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:43.038328 14311 raft_consensus.cc:504] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } }
I20240501 14:09:43.038659 14311 leader_election.cc:288] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e26df472efe547f8bc5951dce352471b (127.7.39.134:33035)
W20240501 14:09:43.040072 13511 leader_election.cc:334] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e26df472efe547f8bc5951dce352471b (127.7.39.134:33035): Network error: Client connection negotiation failed: client connection to 127.7.39.134:33035: connect: Connection refused (error 111)
I20240501 14:09:43.040135 13511 leader_election.cc:302] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 43736346b1ff48c78ac919c80ba58654; no voters: e26df472efe547f8bc5951dce352471b
I20240501 14:09:43.040259 14311 raft_consensus.cc:2726] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:09:43.662845 13303 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43662:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:43.670332 13557 tablet_service.cc:1467] Processing CreateTablet for tablet e4266d76760c4f84b16991f8aa38a3c3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b91b950953be4c81b3156e9c2c4e50ee]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:43.670711 13557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4266d76760c4f84b16991f8aa38a3c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.673092 14322 tablet_bootstrap.cc:492] T e4266d76760c4f84b16991f8aa38a3c3 P 43736346b1ff48c78ac919c80ba58654: Bootstrap starting.
I20240501 14:09:43.674175 14322 tablet_bootstrap.cc:654] T e4266d76760c4f84b16991f8aa38a3c3 P 43736346b1ff48c78ac919c80ba58654: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:43.675992 13425 tablet_service.cc:1467] Processing CreateTablet for tablet bd836361783c4b3782c0f64c6794488b (DEFAULT_TABLE table=rebalance_test_table_0 [id=b91b950953be4c81b3156e9c2c4e50ee]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:43.675997 13687 tablet_service.cc:1467] Processing CreateTablet for tablet bd836361783c4b3782c0f64c6794488b (DEFAULT_TABLE table=rebalance_test_table_0 [id=b91b950953be4c81b3156e9c2c4e50ee]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:43.676393 14322 tablet_bootstrap.cc:492] T e4266d76760c4f84b16991f8aa38a3c3 P 43736346b1ff48c78ac919c80ba58654: No bootstrap required, opened a new log
I20240501 14:09:43.676003 13424 tablet_service.cc:1467] Processing CreateTablet for tablet e4266d76760c4f84b16991f8aa38a3c3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b91b950953be4c81b3156e9c2c4e50ee]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:43.676468 14322 ts_tablet_manager.cc:1367] T e4266d76760c4f84b16991f8aa38a3c3 P 43736346b1ff48c78ac919c80ba58654: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:43.676681 13426 tablet_service.cc:1467] Processing CreateTablet for tablet 341c6b416c374a9ba4fc34d861ba2ef9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b91b950953be4c81b3156e9c2c4e50ee]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:43.676967 13687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd836361783c4b3782c0f64c6794488b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.676967 13424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4266d76760c4f84b16991f8aa38a3c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.677047 14322 raft_consensus.cc:348] T e4266d76760c4f84b16991f8aa38a3c3 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } }
I20240501 14:09:43.677161 14322 raft_consensus.cc:374] T e4266d76760c4f84b16991f8aa38a3c3 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:43.677202 14322 raft_consensus.cc:729] T e4266d76760c4f84b16991f8aa38a3c3 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43736346b1ff48c78ac919c80ba58654, State: Initialized, Role: FOLLOWER
I20240501 14:09:43.675992 13688 tablet_service.cc:1467] Processing CreateTablet for tablet 341c6b416c374a9ba4fc34d861ba2ef9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b91b950953be4c81b3156e9c2c4e50ee]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:43.677335 14322 consensus_queue.cc:260] T e4266d76760c4f84b16991f8aa38a3c3 P 43736346b1ff48c78ac919c80ba58654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } }
I20240501 14:09:43.677424 13688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 341c6b416c374a9ba4fc34d861ba2ef9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.677644 14322 ts_tablet_manager.cc:1398] T e4266d76760c4f84b16991f8aa38a3c3 P 43736346b1ff48c78ac919c80ba58654: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:43.678135 13425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd836361783c4b3782c0f64c6794488b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.678833 13426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 341c6b416c374a9ba4fc34d861ba2ef9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.683522 14326 tablet_bootstrap.cc:492] T bd836361783c4b3782c0f64c6794488b P a278448771e14471ab4df1e6649485bd: Bootstrap starting.
I20240501 14:09:43.684521 14325 tablet_bootstrap.cc:492] T bd836361783c4b3782c0f64c6794488b P 50d514f034204fc38331aa0a4fd74a1b: Bootstrap starting.
I20240501 14:09:43.685258 14326 tablet_bootstrap.cc:654] T bd836361783c4b3782c0f64c6794488b P a278448771e14471ab4df1e6649485bd: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:43.685998 14326 log.cc:824] T bd836361783c4b3782c0f64c6794488b P a278448771e14471ab4df1e6649485bd: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:43.686126 14325 tablet_bootstrap.cc:654] T bd836361783c4b3782c0f64c6794488b P 50d514f034204fc38331aa0a4fd74a1b: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:43.686985 14325 log.cc:824] T bd836361783c4b3782c0f64c6794488b P 50d514f034204fc38331aa0a4fd74a1b: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:43.687595 14326 tablet_bootstrap.cc:492] T bd836361783c4b3782c0f64c6794488b P a278448771e14471ab4df1e6649485bd: No bootstrap required, opened a new log
I20240501 14:09:43.687758 14326 ts_tablet_manager.cc:1367] T bd836361783c4b3782c0f64c6794488b P a278448771e14471ab4df1e6649485bd: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:09:43.688354 14325 tablet_bootstrap.cc:492] T bd836361783c4b3782c0f64c6794488b P 50d514f034204fc38331aa0a4fd74a1b: No bootstrap required, opened a new log
I20240501 14:09:43.688503 14325 ts_tablet_manager.cc:1367] T bd836361783c4b3782c0f64c6794488b P 50d514f034204fc38331aa0a4fd74a1b: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 14:09:43.690474 14326 raft_consensus.cc:348] T bd836361783c4b3782c0f64c6794488b P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } }
I20240501 14:09:43.690706 14326 raft_consensus.cc:374] T bd836361783c4b3782c0f64c6794488b P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:43.690768 14326 raft_consensus.cc:729] T bd836361783c4b3782c0f64c6794488b P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a278448771e14471ab4df1e6649485bd, State: Initialized, Role: FOLLOWER
I20240501 14:09:43.690675 14325 raft_consensus.cc:348] T bd836361783c4b3782c0f64c6794488b P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } }
I20240501 14:09:43.690852 14325 raft_consensus.cc:374] T bd836361783c4b3782c0f64c6794488b P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:43.690901 14325 raft_consensus.cc:729] T bd836361783c4b3782c0f64c6794488b P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50d514f034204fc38331aa0a4fd74a1b, State: Initialized, Role: FOLLOWER
I20240501 14:09:43.691236 14326 consensus_queue.cc:260] T bd836361783c4b3782c0f64c6794488b P a278448771e14471ab4df1e6649485bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } }
I20240501 14:09:43.691313 14325 consensus_queue.cc:260] T bd836361783c4b3782c0f64c6794488b P 50d514f034204fc38331aa0a4fd74a1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } }
I20240501 14:09:43.692018 14326 ts_tablet_manager.cc:1398] T bd836361783c4b3782c0f64c6794488b P a278448771e14471ab4df1e6649485bd: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 14:09:43.692059 14325 ts_tablet_manager.cc:1398] T bd836361783c4b3782c0f64c6794488b P 50d514f034204fc38331aa0a4fd74a1b: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20240501 14:09:43.692289 14326 tablet_bootstrap.cc:492] T 341c6b416c374a9ba4fc34d861ba2ef9 P a278448771e14471ab4df1e6649485bd: Bootstrap starting.
I20240501 14:09:43.692670 14325 tablet_bootstrap.cc:492] T 341c6b416c374a9ba4fc34d861ba2ef9 P 50d514f034204fc38331aa0a4fd74a1b: Bootstrap starting.
I20240501 14:09:43.693221 14326 tablet_bootstrap.cc:654] T 341c6b416c374a9ba4fc34d861ba2ef9 P a278448771e14471ab4df1e6649485bd: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:43.693634 14325 tablet_bootstrap.cc:654] T 341c6b416c374a9ba4fc34d861ba2ef9 P 50d514f034204fc38331aa0a4fd74a1b: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:43.694713 14326 tablet_bootstrap.cc:492] T 341c6b416c374a9ba4fc34d861ba2ef9 P a278448771e14471ab4df1e6649485bd: No bootstrap required, opened a new log
I20240501 14:09:43.694803 14326 ts_tablet_manager.cc:1367] T 341c6b416c374a9ba4fc34d861ba2ef9 P a278448771e14471ab4df1e6649485bd: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:43.695147 14325 tablet_bootstrap.cc:492] T 341c6b416c374a9ba4fc34d861ba2ef9 P 50d514f034204fc38331aa0a4fd74a1b: No bootstrap required, opened a new log
I20240501 14:09:43.695267 14325 ts_tablet_manager.cc:1367] T 341c6b416c374a9ba4fc34d861ba2ef9 P 50d514f034204fc38331aa0a4fd74a1b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:09:43.695413 14326 raft_consensus.cc:348] T 341c6b416c374a9ba4fc34d861ba2ef9 P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } }
I20240501 14:09:43.695513 14326 raft_consensus.cc:374] T 341c6b416c374a9ba4fc34d861ba2ef9 P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:43.695556 14326 raft_consensus.cc:729] T 341c6b416c374a9ba4fc34d861ba2ef9 P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a278448771e14471ab4df1e6649485bd, State: Initialized, Role: FOLLOWER
I20240501 14:09:43.695694 14326 consensus_queue.cc:260] T 341c6b416c374a9ba4fc34d861ba2ef9 P a278448771e14471ab4df1e6649485bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } }
I20240501 14:09:43.695888 14325 raft_consensus.cc:348] T 341c6b416c374a9ba4fc34d861ba2ef9 P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } }
I20240501 14:09:43.695927 14326 ts_tablet_manager.cc:1398] T 341c6b416c374a9ba4fc34d861ba2ef9 P a278448771e14471ab4df1e6649485bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:43.695971 14325 raft_consensus.cc:374] T 341c6b416c374a9ba4fc34d861ba2ef9 P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:43.696012 14325 raft_consensus.cc:729] T 341c6b416c374a9ba4fc34d861ba2ef9 P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50d514f034204fc38331aa0a4fd74a1b, State: Initialized, Role: FOLLOWER
I20240501 14:09:43.696136 14325 consensus_queue.cc:260] T 341c6b416c374a9ba4fc34d861ba2ef9 P 50d514f034204fc38331aa0a4fd74a1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } }
I20240501 14:09:43.696377 14325 ts_tablet_manager.cc:1398] T 341c6b416c374a9ba4fc34d861ba2ef9 P 50d514f034204fc38331aa0a4fd74a1b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:43.696537 14325 tablet_bootstrap.cc:492] T e4266d76760c4f84b16991f8aa38a3c3 P 50d514f034204fc38331aa0a4fd74a1b: Bootstrap starting.
I20240501 14:09:43.697430 14325 tablet_bootstrap.cc:654] T e4266d76760c4f84b16991f8aa38a3c3 P 50d514f034204fc38331aa0a4fd74a1b: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:43.698395 14325 tablet_bootstrap.cc:492] T e4266d76760c4f84b16991f8aa38a3c3 P 50d514f034204fc38331aa0a4fd74a1b: No bootstrap required, opened a new log
I20240501 14:09:43.698467 14325 ts_tablet_manager.cc:1367] T e4266d76760c4f84b16991f8aa38a3c3 P 50d514f034204fc38331aa0a4fd74a1b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:43.698774 14325 raft_consensus.cc:348] T e4266d76760c4f84b16991f8aa38a3c3 P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } }
I20240501 14:09:43.698846 14325 raft_consensus.cc:374] T e4266d76760c4f84b16991f8aa38a3c3 P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:43.698879 14325 raft_consensus.cc:729] T e4266d76760c4f84b16991f8aa38a3c3 P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50d514f034204fc38331aa0a4fd74a1b, State: Initialized, Role: FOLLOWER
I20240501 14:09:43.699008 14325 consensus_queue.cc:260] T e4266d76760c4f84b16991f8aa38a3c3 P 50d514f034204fc38331aa0a4fd74a1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } }
I20240501 14:09:43.699307 14325 ts_tablet_manager.cc:1398] T e4266d76760c4f84b16991f8aa38a3c3 P 50d514f034204fc38331aa0a4fd74a1b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240501 14:09:43.726351 13755 tablet.cc:2369] T 341c6b416c374a9ba4fc34d861ba2ef9 P a278448771e14471ab4df1e6649485bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:09:43.735638 14329 raft_consensus.cc:482] T bd836361783c4b3782c0f64c6794488b P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:43.735826 14329 raft_consensus.cc:504] T bd836361783c4b3782c0f64c6794488b P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } }
I20240501 14:09:43.737013 14329 leader_election.cc:288] T bd836361783c4b3782c0f64c6794488b P a278448771e14471ab4df1e6649485bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129:37933)
I20240501 14:09:43.741166 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd836361783c4b3782c0f64c6794488b" candidate_uuid: "a278448771e14471ab4df1e6649485bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50d514f034204fc38331aa0a4fd74a1b" is_pre_election: true
I20240501 14:09:43.741446 13446 raft_consensus.cc:2454] T bd836361783c4b3782c0f64c6794488b P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a278448771e14471ab4df1e6649485bd in term 0.
I20240501 14:09:43.741875 13644 leader_election.cc:302] T bd836361783c4b3782c0f64c6794488b P a278448771e14471ab4df1e6649485bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 50d514f034204fc38331aa0a4fd74a1b, a278448771e14471ab4df1e6649485bd; no voters:
I20240501 14:09:43.742107 14329 raft_consensus.cc:2781] T bd836361783c4b3782c0f64c6794488b P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:43.742183 14329 raft_consensus.cc:482] T bd836361783c4b3782c0f64c6794488b P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:43.742237 14329 raft_consensus.cc:3037] T bd836361783c4b3782c0f64c6794488b P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:43.743417 14329 raft_consensus.cc:504] T bd836361783c4b3782c0f64c6794488b P a278448771e14471ab4df1e6649485bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } }
I20240501 14:09:43.743711 14329 leader_election.cc:288] T bd836361783c4b3782c0f64c6794488b P a278448771e14471ab4df1e6649485bd [CANDIDATE]: Term 1 election: Requested vote from peers 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129:37933)
I20240501 14:09:43.744036 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd836361783c4b3782c0f64c6794488b" candidate_uuid: "a278448771e14471ab4df1e6649485bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50d514f034204fc38331aa0a4fd74a1b"
I20240501 14:09:43.744140 13446 raft_consensus.cc:3037] T bd836361783c4b3782c0f64c6794488b P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:43.745002 13446 raft_consensus.cc:2454] T bd836361783c4b3782c0f64c6794488b P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a278448771e14471ab4df1e6649485bd in term 1.
I20240501 14:09:43.745256 13644 leader_election.cc:302] T bd836361783c4b3782c0f64c6794488b P a278448771e14471ab4df1e6649485bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 50d514f034204fc38331aa0a4fd74a1b, a278448771e14471ab4df1e6649485bd; no voters:
I20240501 14:09:43.745417 14329 raft_consensus.cc:2781] T bd836361783c4b3782c0f64c6794488b P a278448771e14471ab4df1e6649485bd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:43.745621 14329 raft_consensus.cc:686] T bd836361783c4b3782c0f64c6794488b P a278448771e14471ab4df1e6649485bd [term 1 LEADER]: Becoming Leader. State: Replica: a278448771e14471ab4df1e6649485bd, State: Running, Role: LEADER
I20240501 14:09:43.745919 14329 consensus_queue.cc:237] T bd836361783c4b3782c0f64c6794488b P a278448771e14471ab4df1e6649485bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } }
I20240501 14:09:43.748494 13303 catalog_manager.cc:5484] T bd836361783c4b3782c0f64c6794488b P a278448771e14471ab4df1e6649485bd reported cstate change: term changed from 0 to 1, leader changed from <none> to a278448771e14471ab4df1e6649485bd (127.7.39.131). New cstate: current_term: 1 leader_uuid: "a278448771e14471ab4df1e6649485bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } }
I20240501 14:09:43.809851 14330 raft_consensus.cc:482] T 341c6b416c374a9ba4fc34d861ba2ef9 P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:43.810007 14330 raft_consensus.cc:504] T 341c6b416c374a9ba4fc34d861ba2ef9 P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } }
I20240501 14:09:43.811079 14330 leader_election.cc:288] T 341c6b416c374a9ba4fc34d861ba2ef9 P 50d514f034204fc38331aa0a4fd74a1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a278448771e14471ab4df1e6649485bd (127.7.39.131:46461)
I20240501 14:09:43.814695 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "341c6b416c374a9ba4fc34d861ba2ef9" candidate_uuid: "50d514f034204fc38331aa0a4fd74a1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a278448771e14471ab4df1e6649485bd" is_pre_election: true
I20240501 14:09:43.814885 13708 raft_consensus.cc:2454] T 341c6b416c374a9ba4fc34d861ba2ef9 P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50d514f034204fc38331aa0a4fd74a1b in term 0.
I20240501 14:09:43.815248 13379 leader_election.cc:302] T 341c6b416c374a9ba4fc34d861ba2ef9 P 50d514f034204fc38331aa0a4fd74a1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 50d514f034204fc38331aa0a4fd74a1b, a278448771e14471ab4df1e6649485bd; no voters:
I20240501 14:09:43.815479 14330 raft_consensus.cc:2781] T 341c6b416c374a9ba4fc34d861ba2ef9 P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:43.815541 14330 raft_consensus.cc:482] T 341c6b416c374a9ba4fc34d861ba2ef9 P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:43.815582 14330 raft_consensus.cc:3037] T 341c6b416c374a9ba4fc34d861ba2ef9 P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:43.816421 14330 raft_consensus.cc:504] T 341c6b416c374a9ba4fc34d861ba2ef9 P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } }
I20240501 14:09:43.816711 14330 leader_election.cc:288] T 341c6b416c374a9ba4fc34d861ba2ef9 P 50d514f034204fc38331aa0a4fd74a1b [CANDIDATE]: Term 1 election: Requested vote from peers a278448771e14471ab4df1e6649485bd (127.7.39.131:46461)
I20240501 14:09:43.817106 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "341c6b416c374a9ba4fc34d861ba2ef9" candidate_uuid: "50d514f034204fc38331aa0a4fd74a1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a278448771e14471ab4df1e6649485bd"
I20240501 14:09:43.817206 13708 raft_consensus.cc:3037] T 341c6b416c374a9ba4fc34d861ba2ef9 P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:43.817956 13708 raft_consensus.cc:2454] T 341c6b416c374a9ba4fc34d861ba2ef9 P a278448771e14471ab4df1e6649485bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50d514f034204fc38331aa0a4fd74a1b in term 1.
I20240501 14:09:43.818253 13379 leader_election.cc:302] T 341c6b416c374a9ba4fc34d861ba2ef9 P 50d514f034204fc38331aa0a4fd74a1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 50d514f034204fc38331aa0a4fd74a1b, a278448771e14471ab4df1e6649485bd; no voters:
I20240501 14:09:43.818408 14330 raft_consensus.cc:2781] T 341c6b416c374a9ba4fc34d861ba2ef9 P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:43.818606 14330 raft_consensus.cc:686] T 341c6b416c374a9ba4fc34d861ba2ef9 P 50d514f034204fc38331aa0a4fd74a1b [term 1 LEADER]: Becoming Leader. State: Replica: 50d514f034204fc38331aa0a4fd74a1b, State: Running, Role: LEADER
I20240501 14:09:43.818908 14330 consensus_queue.cc:237] T 341c6b416c374a9ba4fc34d861ba2ef9 P 50d514f034204fc38331aa0a4fd74a1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } }
I20240501 14:09:43.821674 13304 catalog_manager.cc:5484] T 341c6b416c374a9ba4fc34d861ba2ef9 P 50d514f034204fc38331aa0a4fd74a1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129). New cstate: current_term: 1 leader_uuid: "50d514f034204fc38331aa0a4fd74a1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } }
W20240501 14:09:43.870460 13493 tablet.cc:2369] T 341c6b416c374a9ba4fc34d861ba2ef9 P 50d514f034204fc38331aa0a4fd74a1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:09:43.905941 14324 raft_consensus.cc:482] T e4266d76760c4f84b16991f8aa38a3c3 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:43.906087 14324 raft_consensus.cc:504] T e4266d76760c4f84b16991f8aa38a3c3 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } }
I20240501 14:09:43.906574 14324 leader_election.cc:288] T e4266d76760c4f84b16991f8aa38a3c3 P 43736346b1ff48c78ac919c80ba58654 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129:37933)
I20240501 14:09:43.909991 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4266d76760c4f84b16991f8aa38a3c3" candidate_uuid: "43736346b1ff48c78ac919c80ba58654" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50d514f034204fc38331aa0a4fd74a1b" is_pre_election: true
I20240501 14:09:43.910147 13446 raft_consensus.cc:2454] T e4266d76760c4f84b16991f8aa38a3c3 P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43736346b1ff48c78ac919c80ba58654 in term 0.
I20240501 14:09:43.910477 13513 leader_election.cc:302] T e4266d76760c4f84b16991f8aa38a3c3 P 43736346b1ff48c78ac919c80ba58654 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 43736346b1ff48c78ac919c80ba58654, 50d514f034204fc38331aa0a4fd74a1b; no voters:
I20240501 14:09:43.910710 14324 raft_consensus.cc:2781] T e4266d76760c4f84b16991f8aa38a3c3 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:43.910778 14324 raft_consensus.cc:482] T e4266d76760c4f84b16991f8aa38a3c3 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:43.910822 14324 raft_consensus.cc:3037] T e4266d76760c4f84b16991f8aa38a3c3 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:43.911705 14324 raft_consensus.cc:504] T e4266d76760c4f84b16991f8aa38a3c3 P 43736346b1ff48c78ac919c80ba58654 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } }
I20240501 14:09:43.911972 14324 leader_election.cc:288] T e4266d76760c4f84b16991f8aa38a3c3 P 43736346b1ff48c78ac919c80ba58654 [CANDIDATE]: Term 1 election: Requested vote from peers 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129:37933)
I20240501 14:09:43.912328 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4266d76760c4f84b16991f8aa38a3c3" candidate_uuid: "43736346b1ff48c78ac919c80ba58654" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50d514f034204fc38331aa0a4fd74a1b"
I20240501 14:09:43.912427 13446 raft_consensus.cc:3037] T e4266d76760c4f84b16991f8aa38a3c3 P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:43.913146 13446 raft_consensus.cc:2454] T e4266d76760c4f84b16991f8aa38a3c3 P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43736346b1ff48c78ac919c80ba58654 in term 1.
I20240501 14:09:43.913416 13513 leader_election.cc:302] T e4266d76760c4f84b16991f8aa38a3c3 P 43736346b1ff48c78ac919c80ba58654 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 43736346b1ff48c78ac919c80ba58654, 50d514f034204fc38331aa0a4fd74a1b; no voters:
I20240501 14:09:43.913605 14324 raft_consensus.cc:2781] T e4266d76760c4f84b16991f8aa38a3c3 P 43736346b1ff48c78ac919c80ba58654 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:43.913830 14324 raft_consensus.cc:686] T e4266d76760c4f84b16991f8aa38a3c3 P 43736346b1ff48c78ac919c80ba58654 [term 1 LEADER]: Becoming Leader. State: Replica: 43736346b1ff48c78ac919c80ba58654, State: Running, Role: LEADER
I20240501 14:09:43.914145 14324 consensus_queue.cc:237] T e4266d76760c4f84b16991f8aa38a3c3 P 43736346b1ff48c78ac919c80ba58654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } }
I20240501 14:09:43.916699 13303 catalog_manager.cc:5484] T e4266d76760c4f84b16991f8aa38a3c3 P 43736346b1ff48c78ac919c80ba58654 reported cstate change: term changed from 0 to 1, leader changed from <none> to 43736346b1ff48c78ac919c80ba58654 (127.7.39.130). New cstate: current_term: 1 leader_uuid: "43736346b1ff48c78ac919c80ba58654" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } }
W20240501 14:09:44.067855 14343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:44.068111 14343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:44.072368 14343 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240501 14:09:44.159319 13446 raft_consensus.cc:1261] T e4266d76760c4f84b16991f8aa38a3c3 P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Refusing update from remote peer 43736346b1ff48c78ac919c80ba58654: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:44.160079 14324 consensus_queue.cc:1035] T e4266d76760c4f84b16991f8aa38a3c3 P 43736346b1ff48c78ac919c80ba58654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:44.161379 13708 raft_consensus.cc:1261] T 341c6b416c374a9ba4fc34d861ba2ef9 P a278448771e14471ab4df1e6649485bd [term 1 FOLLOWER]: Refusing update from remote peer 50d514f034204fc38331aa0a4fd74a1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:44.161778 13446 raft_consensus.cc:1261] T bd836361783c4b3782c0f64c6794488b P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Refusing update from remote peer a278448771e14471ab4df1e6649485bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:44.161932 14330 consensus_queue.cc:1035] T 341c6b416c374a9ba4fc34d861ba2ef9 P 50d514f034204fc38331aa0a4fd74a1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:44.164422 14329 consensus_queue.cc:1035] T bd836361783c4b3782c0f64c6794488b P a278448771e14471ab4df1e6649485bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:44.164640 14378 mvcc.cc:205] Tried to move back new op lower bound from 7022889304708988928 to 7022889303714672640. Current Snapshot: MvccSnapshot[applied={T|T < 7022889304708988928}]
I20240501 14:09:44.171177 14376 mvcc.cc:205] Tried to move back new op lower bound from 7022889304723488768 to 7022889303025451008. Current Snapshot: MvccSnapshot[applied={T|T < 7022889304708497408}]
I20240501 14:09:44.173470 14379 mvcc.cc:205] Tried to move back new op lower bound from 7022889304723488768 to 7022889303025451008. Current Snapshot: MvccSnapshot[applied={T|T < 7022889304708497408}]
INSERT report
rows total: 2000
time total: 87.6396 ms
time per row: 0.0438198 ms
I20240501 14:09:44.203244 13304 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43662:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:44.208462 13688 tablet_service.cc:1467] Processing CreateTablet for tablet 8a89cc8e4b2b476ba954d3cc64537031 (DEFAULT_TABLE table=rebalance_test_table_1 [id=b18f113d94ba4379ab99c6c63917bc1a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:44.208621 13557 tablet_service.cc:1467] Processing CreateTablet for tablet 8a89cc8e4b2b476ba954d3cc64537031 (DEFAULT_TABLE table=rebalance_test_table_1 [id=b18f113d94ba4379ab99c6c63917bc1a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:44.208820 13688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a89cc8e4b2b476ba954d3cc64537031. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:44.208856 13557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a89cc8e4b2b476ba954d3cc64537031. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:44.209261 13424 tablet_service.cc:1467] Processing CreateTablet for tablet 3f34112bba8a43a58420148533a6e989 (DEFAULT_TABLE table=rebalance_test_table_1 [id=b18f113d94ba4379ab99c6c63917bc1a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:44.209483 13424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f34112bba8a43a58420148533a6e989. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:44.211177 13556 tablet_service.cc:1467] Processing CreateTablet for tablet d948c2d5182b415b8aedb4c7b786247f (DEFAULT_TABLE table=rebalance_test_table_1 [id=b18f113d94ba4379ab99c6c63917bc1a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:44.211441 13556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d948c2d5182b415b8aedb4c7b786247f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:44.211637 14393 tablet_bootstrap.cc:492] T 8a89cc8e4b2b476ba954d3cc64537031 P a278448771e14471ab4df1e6649485bd: Bootstrap starting.
I20240501 14:09:44.212435 14393 tablet_bootstrap.cc:654] T 8a89cc8e4b2b476ba954d3cc64537031 P a278448771e14471ab4df1e6649485bd: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:44.213063 14394 tablet_bootstrap.cc:492] T 8a89cc8e4b2b476ba954d3cc64537031 P 43736346b1ff48c78ac919c80ba58654: Bootstrap starting.
I20240501 14:09:44.213191 13555 tablet_service.cc:1467] Processing CreateTablet for tablet 3f34112bba8a43a58420148533a6e989 (DEFAULT_TABLE table=rebalance_test_table_1 [id=b18f113d94ba4379ab99c6c63917bc1a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:44.213438 14393 tablet_bootstrap.cc:492] T 8a89cc8e4b2b476ba954d3cc64537031 P a278448771e14471ab4df1e6649485bd: No bootstrap required, opened a new log
I20240501 14:09:44.213526 14393 ts_tablet_manager.cc:1367] T 8a89cc8e4b2b476ba954d3cc64537031 P a278448771e14471ab4df1e6649485bd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:44.213860 13688 tablet_service.cc:1467] Processing CreateTablet for tablet d948c2d5182b415b8aedb4c7b786247f (DEFAULT_TABLE table=rebalance_test_table_1 [id=b18f113d94ba4379ab99c6c63917bc1a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:44.213990 14393 raft_consensus.cc:348] T 8a89cc8e4b2b476ba954d3cc64537031 P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } }
I20240501 14:09:44.214095 13688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d948c2d5182b415b8aedb4c7b786247f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:44.214280 14395 tablet_bootstrap.cc:492] T 3f34112bba8a43a58420148533a6e989 P 50d514f034204fc38331aa0a4fd74a1b: Bootstrap starting.
I20240501 14:09:44.215230 14395 tablet_bootstrap.cc:654] T 3f34112bba8a43a58420148533a6e989 P 50d514f034204fc38331aa0a4fd74a1b: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:44.214110 14393 raft_consensus.cc:374] T 8a89cc8e4b2b476ba954d3cc64537031 P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:44.215705 14393 raft_consensus.cc:729] T 8a89cc8e4b2b476ba954d3cc64537031 P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a278448771e14471ab4df1e6649485bd, State: Initialized, Role: FOLLOWER
I20240501 14:09:44.215845 14393 consensus_queue.cc:260] T 8a89cc8e4b2b476ba954d3cc64537031 P a278448771e14471ab4df1e6649485bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } }
I20240501 14:09:44.216125 14393 ts_tablet_manager.cc:1398] T 8a89cc8e4b2b476ba954d3cc64537031 P a278448771e14471ab4df1e6649485bd: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:44.216280 14393 tablet_bootstrap.cc:492] T d948c2d5182b415b8aedb4c7b786247f P a278448771e14471ab4df1e6649485bd: Bootstrap starting.
I20240501 14:09:44.216449 14395 tablet_bootstrap.cc:492] T 3f34112bba8a43a58420148533a6e989 P 50d514f034204fc38331aa0a4fd74a1b: No bootstrap required, opened a new log
I20240501 14:09:44.216554 14395 ts_tablet_manager.cc:1367] T 3f34112bba8a43a58420148533a6e989 P 50d514f034204fc38331aa0a4fd74a1b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:44.216995 14395 raft_consensus.cc:348] T 3f34112bba8a43a58420148533a6e989 P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } }
I20240501 14:09:44.217115 14395 raft_consensus.cc:374] T 3f34112bba8a43a58420148533a6e989 P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:44.217146 14395 raft_consensus.cc:729] T 3f34112bba8a43a58420148533a6e989 P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50d514f034204fc38331aa0a4fd74a1b, State: Initialized, Role: FOLLOWER
I20240501 14:09:44.217267 14395 consensus_queue.cc:260] T 3f34112bba8a43a58420148533a6e989 P 50d514f034204fc38331aa0a4fd74a1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } }
I20240501 14:09:44.217554 14395 ts_tablet_manager.cc:1398] T 3f34112bba8a43a58420148533a6e989 P 50d514f034204fc38331aa0a4fd74a1b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:44.217715 14393 tablet_bootstrap.cc:654] T d948c2d5182b415b8aedb4c7b786247f P a278448771e14471ab4df1e6649485bd: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:44.218349 13555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f34112bba8a43a58420148533a6e989. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:44.219182 14394 tablet_bootstrap.cc:654] T 8a89cc8e4b2b476ba954d3cc64537031 P 43736346b1ff48c78ac919c80ba58654: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:44.220108 14393 tablet_bootstrap.cc:492] T d948c2d5182b415b8aedb4c7b786247f P a278448771e14471ab4df1e6649485bd: No bootstrap required, opened a new log
I20240501 14:09:44.220185 14393 ts_tablet_manager.cc:1367] T d948c2d5182b415b8aedb4c7b786247f P a278448771e14471ab4df1e6649485bd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:44.220635 14393 raft_consensus.cc:348] T d948c2d5182b415b8aedb4c7b786247f P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } }
I20240501 14:09:44.220743 14393 raft_consensus.cc:374] T d948c2d5182b415b8aedb4c7b786247f P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:44.220796 14393 raft_consensus.cc:729] T d948c2d5182b415b8aedb4c7b786247f P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a278448771e14471ab4df1e6649485bd, State: Initialized, Role: FOLLOWER
I20240501 14:09:44.220916 14393 consensus_queue.cc:260] T d948c2d5182b415b8aedb4c7b786247f P a278448771e14471ab4df1e6649485bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } }
I20240501 14:09:44.221216 14393 ts_tablet_manager.cc:1398] T d948c2d5182b415b8aedb4c7b786247f P a278448771e14471ab4df1e6649485bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:44.221668 14394 tablet_bootstrap.cc:492] T 8a89cc8e4b2b476ba954d3cc64537031 P 43736346b1ff48c78ac919c80ba58654: No bootstrap required, opened a new log
I20240501 14:09:44.221763 14394 ts_tablet_manager.cc:1367] T 8a89cc8e4b2b476ba954d3cc64537031 P 43736346b1ff48c78ac919c80ba58654: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.001s
I20240501 14:09:44.222237 14394 raft_consensus.cc:348] T 8a89cc8e4b2b476ba954d3cc64537031 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } }
I20240501 14:09:44.222349 14394 raft_consensus.cc:374] T 8a89cc8e4b2b476ba954d3cc64537031 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:44.222391 14394 raft_consensus.cc:729] T 8a89cc8e4b2b476ba954d3cc64537031 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43736346b1ff48c78ac919c80ba58654, State: Initialized, Role: FOLLOWER
I20240501 14:09:44.222529 14394 consensus_queue.cc:260] T 8a89cc8e4b2b476ba954d3cc64537031 P 43736346b1ff48c78ac919c80ba58654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } }
I20240501 14:09:44.222785 14394 ts_tablet_manager.cc:1398] T 8a89cc8e4b2b476ba954d3cc64537031 P 43736346b1ff48c78ac919c80ba58654: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:44.222965 14394 tablet_bootstrap.cc:492] T d948c2d5182b415b8aedb4c7b786247f P 43736346b1ff48c78ac919c80ba58654: Bootstrap starting.
I20240501 14:09:44.223865 14394 tablet_bootstrap.cc:654] T d948c2d5182b415b8aedb4c7b786247f P 43736346b1ff48c78ac919c80ba58654: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:44.224992 14394 tablet_bootstrap.cc:492] T d948c2d5182b415b8aedb4c7b786247f P 43736346b1ff48c78ac919c80ba58654: No bootstrap required, opened a new log
I20240501 14:09:44.225088 14394 ts_tablet_manager.cc:1367] T d948c2d5182b415b8aedb4c7b786247f P 43736346b1ff48c78ac919c80ba58654: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:44.225503 14394 raft_consensus.cc:348] T d948c2d5182b415b8aedb4c7b786247f P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } }
I20240501 14:09:44.225600 14394 raft_consensus.cc:374] T d948c2d5182b415b8aedb4c7b786247f P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:44.225633 14394 raft_consensus.cc:729] T d948c2d5182b415b8aedb4c7b786247f P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43736346b1ff48c78ac919c80ba58654, State: Initialized, Role: FOLLOWER
I20240501 14:09:44.225762 14394 consensus_queue.cc:260] T d948c2d5182b415b8aedb4c7b786247f P 43736346b1ff48c78ac919c80ba58654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } }
I20240501 14:09:44.226017 14394 ts_tablet_manager.cc:1398] T d948c2d5182b415b8aedb4c7b786247f P 43736346b1ff48c78ac919c80ba58654: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:44.226202 14394 tablet_bootstrap.cc:492] T 3f34112bba8a43a58420148533a6e989 P 43736346b1ff48c78ac919c80ba58654: Bootstrap starting.
I20240501 14:09:44.226871 14394 tablet_bootstrap.cc:654] T 3f34112bba8a43a58420148533a6e989 P 43736346b1ff48c78ac919c80ba58654: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:44.228159 14394 tablet_bootstrap.cc:492] T 3f34112bba8a43a58420148533a6e989 P 43736346b1ff48c78ac919c80ba58654: No bootstrap required, opened a new log
I20240501 14:09:44.228237 14394 ts_tablet_manager.cc:1367] T 3f34112bba8a43a58420148533a6e989 P 43736346b1ff48c78ac919c80ba58654: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:44.228545 14394 raft_consensus.cc:348] T 3f34112bba8a43a58420148533a6e989 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } }
I20240501 14:09:44.228622 14394 raft_consensus.cc:374] T 3f34112bba8a43a58420148533a6e989 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:44.228655 14394 raft_consensus.cc:729] T 3f34112bba8a43a58420148533a6e989 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43736346b1ff48c78ac919c80ba58654, State: Initialized, Role: FOLLOWER
I20240501 14:09:44.228785 14394 consensus_queue.cc:260] T 3f34112bba8a43a58420148533a6e989 P 43736346b1ff48c78ac919c80ba58654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } }
I20240501 14:09:44.229008 14394 ts_tablet_manager.cc:1398] T 3f34112bba8a43a58420148533a6e989 P 43736346b1ff48c78ac919c80ba58654: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:44.260421 14335 raft_consensus.cc:482] T d948c2d5182b415b8aedb4c7b786247f P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:44.260564 14335 raft_consensus.cc:504] T d948c2d5182b415b8aedb4c7b786247f P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } }
I20240501 14:09:44.261005 14335 leader_election.cc:288] T d948c2d5182b415b8aedb4c7b786247f P a278448771e14471ab4df1e6649485bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43736346b1ff48c78ac919c80ba58654 (127.7.39.130:43311)
I20240501 14:09:44.264678 13577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d948c2d5182b415b8aedb4c7b786247f" candidate_uuid: "a278448771e14471ab4df1e6649485bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43736346b1ff48c78ac919c80ba58654" is_pre_election: true
I20240501 14:09:44.264848 13577 raft_consensus.cc:2454] T d948c2d5182b415b8aedb4c7b786247f P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a278448771e14471ab4df1e6649485bd in term 0.
I20240501 14:09:44.265226 13644 leader_election.cc:302] T d948c2d5182b415b8aedb4c7b786247f P a278448771e14471ab4df1e6649485bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 43736346b1ff48c78ac919c80ba58654, a278448771e14471ab4df1e6649485bd; no voters:
I20240501 14:09:44.265419 14335 raft_consensus.cc:2781] T d948c2d5182b415b8aedb4c7b786247f P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:44.265483 14335 raft_consensus.cc:482] T d948c2d5182b415b8aedb4c7b786247f P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:44.265515 14335 raft_consensus.cc:3037] T d948c2d5182b415b8aedb4c7b786247f P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:44.267796 14335 raft_consensus.cc:504] T d948c2d5182b415b8aedb4c7b786247f P a278448771e14471ab4df1e6649485bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } }
I20240501 14:09:44.268116 14335 leader_election.cc:288] T d948c2d5182b415b8aedb4c7b786247f P a278448771e14471ab4df1e6649485bd [CANDIDATE]: Term 1 election: Requested vote from peers 43736346b1ff48c78ac919c80ba58654 (127.7.39.130:43311)
I20240501 14:09:44.268486 13577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d948c2d5182b415b8aedb4c7b786247f" candidate_uuid: "a278448771e14471ab4df1e6649485bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43736346b1ff48c78ac919c80ba58654"
I20240501 14:09:44.268632 13577 raft_consensus.cc:3037] T d948c2d5182b415b8aedb4c7b786247f P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:44.269419 13577 raft_consensus.cc:2454] T d948c2d5182b415b8aedb4c7b786247f P 43736346b1ff48c78ac919c80ba58654 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a278448771e14471ab4df1e6649485bd in term 1.
I20240501 14:09:44.269763 13644 leader_election.cc:302] T d948c2d5182b415b8aedb4c7b786247f P a278448771e14471ab4df1e6649485bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 43736346b1ff48c78ac919c80ba58654, a278448771e14471ab4df1e6649485bd; no voters:
I20240501 14:09:44.269929 14335 raft_consensus.cc:2781] T d948c2d5182b415b8aedb4c7b786247f P a278448771e14471ab4df1e6649485bd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:44.270035 14335 raft_consensus.cc:686] T d948c2d5182b415b8aedb4c7b786247f P a278448771e14471ab4df1e6649485bd [term 1 LEADER]: Becoming Leader. State: Replica: a278448771e14471ab4df1e6649485bd, State: Running, Role: LEADER
I20240501 14:09:44.270190 14335 consensus_queue.cc:237] T d948c2d5182b415b8aedb4c7b786247f P a278448771e14471ab4df1e6649485bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } }
I20240501 14:09:44.271886 13301 catalog_manager.cc:5484] T d948c2d5182b415b8aedb4c7b786247f P a278448771e14471ab4df1e6649485bd reported cstate change: term changed from 0 to 1, leader changed from <none> to a278448771e14471ab4df1e6649485bd (127.7.39.131). New cstate: current_term: 1 leader_uuid: "a278448771e14471ab4df1e6649485bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } }
I20240501 14:09:44.289505 14338 raft_consensus.cc:482] T 3f34112bba8a43a58420148533a6e989 P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:44.289656 14338 raft_consensus.cc:504] T 3f34112bba8a43a58420148533a6e989 P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } }
I20240501 14:09:44.290016 14338 leader_election.cc:288] T 3f34112bba8a43a58420148533a6e989 P 50d514f034204fc38331aa0a4fd74a1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43736346b1ff48c78ac919c80ba58654 (127.7.39.130:43311)
I20240501 14:09:44.293167 13577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f34112bba8a43a58420148533a6e989" candidate_uuid: "50d514f034204fc38331aa0a4fd74a1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43736346b1ff48c78ac919c80ba58654" is_pre_election: true
I20240501 14:09:44.293305 13577 raft_consensus.cc:2454] T 3f34112bba8a43a58420148533a6e989 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50d514f034204fc38331aa0a4fd74a1b in term 0.
I20240501 14:09:44.293627 13382 leader_election.cc:302] T 3f34112bba8a43a58420148533a6e989 P 50d514f034204fc38331aa0a4fd74a1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 43736346b1ff48c78ac919c80ba58654, 50d514f034204fc38331aa0a4fd74a1b; no voters:
I20240501 14:09:44.293804 14338 raft_consensus.cc:2781] T 3f34112bba8a43a58420148533a6e989 P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:44.293867 14338 raft_consensus.cc:482] T 3f34112bba8a43a58420148533a6e989 P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:44.293915 14338 raft_consensus.cc:3037] T 3f34112bba8a43a58420148533a6e989 P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:44.294770 14338 raft_consensus.cc:504] T 3f34112bba8a43a58420148533a6e989 P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } }
I20240501 14:09:44.295032 14338 leader_election.cc:288] T 3f34112bba8a43a58420148533a6e989 P 50d514f034204fc38331aa0a4fd74a1b [CANDIDATE]: Term 1 election: Requested vote from peers 43736346b1ff48c78ac919c80ba58654 (127.7.39.130:43311)
I20240501 14:09:44.295360 13577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f34112bba8a43a58420148533a6e989" candidate_uuid: "50d514f034204fc38331aa0a4fd74a1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43736346b1ff48c78ac919c80ba58654"
I20240501 14:09:44.295464 13577 raft_consensus.cc:3037] T 3f34112bba8a43a58420148533a6e989 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:44.296224 13577 raft_consensus.cc:2454] T 3f34112bba8a43a58420148533a6e989 P 43736346b1ff48c78ac919c80ba58654 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50d514f034204fc38331aa0a4fd74a1b in term 1.
I20240501 14:09:44.296535 13382 leader_election.cc:302] T 3f34112bba8a43a58420148533a6e989 P 50d514f034204fc38331aa0a4fd74a1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 43736346b1ff48c78ac919c80ba58654, 50d514f034204fc38331aa0a4fd74a1b; no voters:
I20240501 14:09:44.296694 14338 raft_consensus.cc:2781] T 3f34112bba8a43a58420148533a6e989 P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:44.296818 14338 raft_consensus.cc:686] T 3f34112bba8a43a58420148533a6e989 P 50d514f034204fc38331aa0a4fd74a1b [term 1 LEADER]: Becoming Leader. State: Replica: 50d514f034204fc38331aa0a4fd74a1b, State: Running, Role: LEADER
I20240501 14:09:44.296955 14338 consensus_queue.cc:237] T 3f34112bba8a43a58420148533a6e989 P 50d514f034204fc38331aa0a4fd74a1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } }
I20240501 14:09:44.298347 13301 catalog_manager.cc:5484] T 3f34112bba8a43a58420148533a6e989 P 50d514f034204fc38331aa0a4fd74a1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129). New cstate: current_term: 1 leader_uuid: "50d514f034204fc38331aa0a4fd74a1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } }
I20240501 14:09:44.422156 14341 raft_consensus.cc:482] T 8a89cc8e4b2b476ba954d3cc64537031 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:44.422313 14341 raft_consensus.cc:504] T 8a89cc8e4b2b476ba954d3cc64537031 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } }
I20240501 14:09:44.422747 14341 leader_election.cc:288] T 8a89cc8e4b2b476ba954d3cc64537031 P 43736346b1ff48c78ac919c80ba58654 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a278448771e14471ab4df1e6649485bd (127.7.39.131:46461)
I20240501 14:09:44.426470 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a89cc8e4b2b476ba954d3cc64537031" candidate_uuid: "43736346b1ff48c78ac919c80ba58654" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a278448771e14471ab4df1e6649485bd" is_pre_election: true
I20240501 14:09:44.426631 13708 raft_consensus.cc:2454] T 8a89cc8e4b2b476ba954d3cc64537031 P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43736346b1ff48c78ac919c80ba58654 in term 0.
I20240501 14:09:44.426932 13510 leader_election.cc:302] T 8a89cc8e4b2b476ba954d3cc64537031 P 43736346b1ff48c78ac919c80ba58654 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 43736346b1ff48c78ac919c80ba58654, a278448771e14471ab4df1e6649485bd; no voters:
I20240501 14:09:44.427160 14341 raft_consensus.cc:2781] T 8a89cc8e4b2b476ba954d3cc64537031 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:44.427217 14341 raft_consensus.cc:482] T 8a89cc8e4b2b476ba954d3cc64537031 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:44.427276 14341 raft_consensus.cc:3037] T 8a89cc8e4b2b476ba954d3cc64537031 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:44.428180 14341 raft_consensus.cc:504] T 8a89cc8e4b2b476ba954d3cc64537031 P 43736346b1ff48c78ac919c80ba58654 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } }
I20240501 14:09:44.428452 14341 leader_election.cc:288] T 8a89cc8e4b2b476ba954d3cc64537031 P 43736346b1ff48c78ac919c80ba58654 [CANDIDATE]: Term 1 election: Requested vote from peers a278448771e14471ab4df1e6649485bd (127.7.39.131:46461)
I20240501 14:09:44.428808 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a89cc8e4b2b476ba954d3cc64537031" candidate_uuid: "43736346b1ff48c78ac919c80ba58654" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a278448771e14471ab4df1e6649485bd"
I20240501 14:09:44.428912 13708 raft_consensus.cc:3037] T 8a89cc8e4b2b476ba954d3cc64537031 P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:44.429651 13708 raft_consensus.cc:2454] T 8a89cc8e4b2b476ba954d3cc64537031 P a278448771e14471ab4df1e6649485bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43736346b1ff48c78ac919c80ba58654 in term 1.
I20240501 14:09:44.429935 13510 leader_election.cc:302] T 8a89cc8e4b2b476ba954d3cc64537031 P 43736346b1ff48c78ac919c80ba58654 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 43736346b1ff48c78ac919c80ba58654, a278448771e14471ab4df1e6649485bd; no voters:
I20240501 14:09:44.430094 14341 raft_consensus.cc:2781] T 8a89cc8e4b2b476ba954d3cc64537031 P 43736346b1ff48c78ac919c80ba58654 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:44.430189 14341 raft_consensus.cc:686] T 8a89cc8e4b2b476ba954d3cc64537031 P 43736346b1ff48c78ac919c80ba58654 [term 1 LEADER]: Becoming Leader. State: Replica: 43736346b1ff48c78ac919c80ba58654, State: Running, Role: LEADER
I20240501 14:09:44.430328 14341 consensus_queue.cc:237] T 8a89cc8e4b2b476ba954d3cc64537031 P 43736346b1ff48c78ac919c80ba58654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } }
I20240501 14:09:44.431836 13301 catalog_manager.cc:5484] T 8a89cc8e4b2b476ba954d3cc64537031 P 43736346b1ff48c78ac919c80ba58654 reported cstate change: term changed from 0 to 1, leader changed from <none> to 43736346b1ff48c78ac919c80ba58654 (127.7.39.130). New cstate: current_term: 1 leader_uuid: "43736346b1ff48c78ac919c80ba58654" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } }
W20240501 14:09:44.548503 14407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:44.548842 14407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:44.552999 14407 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 14:09:44.631685 13708 raft_consensus.cc:1261] T 8a89cc8e4b2b476ba954d3cc64537031 P a278448771e14471ab4df1e6649485bd [term 1 FOLLOWER]: Refusing update from remote peer 43736346b1ff48c78ac919c80ba58654: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:44.632074 14341 consensus_queue.cc:1035] T 8a89cc8e4b2b476ba954d3cc64537031 P 43736346b1ff48c78ac919c80ba58654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:44.633759 13577 raft_consensus.cc:1261] T 3f34112bba8a43a58420148533a6e989 P 43736346b1ff48c78ac919c80ba58654 [term 1 FOLLOWER]: Refusing update from remote peer 50d514f034204fc38331aa0a4fd74a1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:44.634493 14338 consensus_queue.cc:1035] T 3f34112bba8a43a58420148533a6e989 P 50d514f034204fc38331aa0a4fd74a1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:44.635314 13577 raft_consensus.cc:1261] T d948c2d5182b415b8aedb4c7b786247f P 43736346b1ff48c78ac919c80ba58654 [term 1 FOLLOWER]: Refusing update from remote peer a278448771e14471ab4df1e6649485bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:44.636049 14329 consensus_queue.cc:1035] T d948c2d5182b415b8aedb4c7b786247f P a278448771e14471ab4df1e6649485bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 80.3174 ms
time per row: 0.0401587 ms
I20240501 14:09:44.676033 13303 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43662:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:44.681494 13424 tablet_service.cc:1467] Processing CreateTablet for tablet ba183686332c4ab5b606bbc5f1b4e787 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c8e8153727f143bdae4e267482cc7660]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:44.681648 13688 tablet_service.cc:1467] Processing CreateTablet for tablet ba183686332c4ab5b606bbc5f1b4e787 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c8e8153727f143bdae4e267482cc7660]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:44.681840 13424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba183686332c4ab5b606bbc5f1b4e787. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:44.681907 13688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba183686332c4ab5b606bbc5f1b4e787. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:44.682153 13426 tablet_service.cc:1467] Processing CreateTablet for tablet 0dc94f932e9a4319988e700aff042f22 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c8e8153727f143bdae4e267482cc7660]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:44.682155 13555 tablet_service.cc:1467] Processing CreateTablet for tablet 0dc94f932e9a4319988e700aff042f22 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c8e8153727f143bdae4e267482cc7660]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:44.682404 13426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dc94f932e9a4319988e700aff042f22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:44.682404 13555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dc94f932e9a4319988e700aff042f22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:44.684144 13425 tablet_service.cc:1467] Processing CreateTablet for tablet 4d3b3213e1dd426b8efc3fdc733b1842 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c8e8153727f143bdae4e267482cc7660]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:44.684145 13687 tablet_service.cc:1467] Processing CreateTablet for tablet 4d3b3213e1dd426b8efc3fdc733b1842 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c8e8153727f143bdae4e267482cc7660]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:44.684352 13687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d3b3213e1dd426b8efc3fdc733b1842. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:44.684352 13425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d3b3213e1dd426b8efc3fdc733b1842. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:44.684348 14395 tablet_bootstrap.cc:492] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b: Bootstrap starting.
I20240501 14:09:44.685178 14395 tablet_bootstrap.cc:654] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:44.686208 14394 tablet_bootstrap.cc:492] T 0dc94f932e9a4319988e700aff042f22 P 43736346b1ff48c78ac919c80ba58654: Bootstrap starting.
I20240501 14:09:44.686592 14395 tablet_bootstrap.cc:492] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b: No bootstrap required, opened a new log
I20240501 14:09:44.686679 14395 ts_tablet_manager.cc:1367] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:44.686959 14394 tablet_bootstrap.cc:654] T 0dc94f932e9a4319988e700aff042f22 P 43736346b1ff48c78ac919c80ba58654: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:44.687105 14395 raft_consensus.cc:348] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } }
I20240501 14:09:44.687201 14395 raft_consensus.cc:374] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:44.687247 14395 raft_consensus.cc:729] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50d514f034204fc38331aa0a4fd74a1b, State: Initialized, Role: FOLLOWER
I20240501 14:09:44.687383 14395 consensus_queue.cc:260] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } }
I20240501 14:09:44.687615 14393 tablet_bootstrap.cc:492] T ba183686332c4ab5b606bbc5f1b4e787 P a278448771e14471ab4df1e6649485bd: Bootstrap starting.
I20240501 14:09:44.687649 14395 ts_tablet_manager.cc:1398] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:44.687830 14395 tablet_bootstrap.cc:492] T 0dc94f932e9a4319988e700aff042f22 P 50d514f034204fc38331aa0a4fd74a1b: Bootstrap starting.
I20240501 14:09:44.688066 14394 tablet_bootstrap.cc:492] T 0dc94f932e9a4319988e700aff042f22 P 43736346b1ff48c78ac919c80ba58654: No bootstrap required, opened a new log
I20240501 14:09:44.688153 14394 ts_tablet_manager.cc:1367] T 0dc94f932e9a4319988e700aff042f22 P 43736346b1ff48c78ac919c80ba58654: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:44.688491 14393 tablet_bootstrap.cc:654] T ba183686332c4ab5b606bbc5f1b4e787 P a278448771e14471ab4df1e6649485bd: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:44.688644 14395 tablet_bootstrap.cc:654] T 0dc94f932e9a4319988e700aff042f22 P 50d514f034204fc38331aa0a4fd74a1b: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:44.688606 14394 raft_consensus.cc:348] T 0dc94f932e9a4319988e700aff042f22 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } }
I20240501 14:09:44.688702 14394 raft_consensus.cc:374] T 0dc94f932e9a4319988e700aff042f22 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:44.688737 14394 raft_consensus.cc:729] T 0dc94f932e9a4319988e700aff042f22 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43736346b1ff48c78ac919c80ba58654, State: Initialized, Role: FOLLOWER
I20240501 14:09:44.688853 14394 consensus_queue.cc:260] T 0dc94f932e9a4319988e700aff042f22 P 43736346b1ff48c78ac919c80ba58654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } }
I20240501 14:09:44.689122 14394 ts_tablet_manager.cc:1398] T 0dc94f932e9a4319988e700aff042f22 P 43736346b1ff48c78ac919c80ba58654: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:44.689865 14395 tablet_bootstrap.cc:492] T 0dc94f932e9a4319988e700aff042f22 P 50d514f034204fc38331aa0a4fd74a1b: No bootstrap required, opened a new log
I20240501 14:09:44.689867 14393 tablet_bootstrap.cc:492] T ba183686332c4ab5b606bbc5f1b4e787 P a278448771e14471ab4df1e6649485bd: No bootstrap required, opened a new log
I20240501 14:09:44.689949 14393 ts_tablet_manager.cc:1367] T ba183686332c4ab5b606bbc5f1b4e787 P a278448771e14471ab4df1e6649485bd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:44.689949 14395 ts_tablet_manager.cc:1367] T 0dc94f932e9a4319988e700aff042f22 P 50d514f034204fc38331aa0a4fd74a1b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:44.690361 14395 raft_consensus.cc:348] T 0dc94f932e9a4319988e700aff042f22 P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } }
I20240501 14:09:44.690361 14393 raft_consensus.cc:348] T ba183686332c4ab5b606bbc5f1b4e787 P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } }
I20240501 14:09:44.690451 14395 raft_consensus.cc:374] T 0dc94f932e9a4319988e700aff042f22 P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:44.690451 14393 raft_consensus.cc:374] T ba183686332c4ab5b606bbc5f1b4e787 P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:44.690487 14395 raft_consensus.cc:729] T 0dc94f932e9a4319988e700aff042f22 P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50d514f034204fc38331aa0a4fd74a1b, State: Initialized, Role: FOLLOWER
I20240501 14:09:44.690496 14393 raft_consensus.cc:729] T ba183686332c4ab5b606bbc5f1b4e787 P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a278448771e14471ab4df1e6649485bd, State: Initialized, Role: FOLLOWER
I20240501 14:09:44.690601 14393 consensus_queue.cc:260] T ba183686332c4ab5b606bbc5f1b4e787 P a278448771e14471ab4df1e6649485bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } }
I20240501 14:09:44.690603 14395 consensus_queue.cc:260] T 0dc94f932e9a4319988e700aff042f22 P 50d514f034204fc38331aa0a4fd74a1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } }
I20240501 14:09:44.690831 14393 ts_tablet_manager.cc:1398] T ba183686332c4ab5b606bbc5f1b4e787 P a278448771e14471ab4df1e6649485bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:44.690831 14395 ts_tablet_manager.cc:1398] T 0dc94f932e9a4319988e700aff042f22 P 50d514f034204fc38331aa0a4fd74a1b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:44.690968 14393 tablet_bootstrap.cc:492] T 4d3b3213e1dd426b8efc3fdc733b1842 P a278448771e14471ab4df1e6649485bd: Bootstrap starting.
I20240501 14:09:44.690975 14395 tablet_bootstrap.cc:492] T 4d3b3213e1dd426b8efc3fdc733b1842 P 50d514f034204fc38331aa0a4fd74a1b: Bootstrap starting.
I20240501 14:09:44.691776 14395 tablet_bootstrap.cc:654] T 4d3b3213e1dd426b8efc3fdc733b1842 P 50d514f034204fc38331aa0a4fd74a1b: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:44.692792 14393 tablet_bootstrap.cc:654] T 4d3b3213e1dd426b8efc3fdc733b1842 P a278448771e14471ab4df1e6649485bd: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:44.693203 14395 tablet_bootstrap.cc:492] T 4d3b3213e1dd426b8efc3fdc733b1842 P 50d514f034204fc38331aa0a4fd74a1b: No bootstrap required, opened a new log
I20240501 14:09:44.693289 14395 ts_tablet_manager.cc:1367] T 4d3b3213e1dd426b8efc3fdc733b1842 P 50d514f034204fc38331aa0a4fd74a1b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:44.693950 14395 raft_consensus.cc:348] T 4d3b3213e1dd426b8efc3fdc733b1842 P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } }
I20240501 14:09:44.694061 14395 raft_consensus.cc:374] T 4d3b3213e1dd426b8efc3fdc733b1842 P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:44.694104 14395 raft_consensus.cc:729] T 4d3b3213e1dd426b8efc3fdc733b1842 P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50d514f034204fc38331aa0a4fd74a1b, State: Initialized, Role: FOLLOWER
I20240501 14:09:44.694204 14395 consensus_queue.cc:260] T 4d3b3213e1dd426b8efc3fdc733b1842 P 50d514f034204fc38331aa0a4fd74a1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } }
I20240501 14:09:44.694528 14395 ts_tablet_manager.cc:1398] T 4d3b3213e1dd426b8efc3fdc733b1842 P 50d514f034204fc38331aa0a4fd74a1b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:44.695106 14393 tablet_bootstrap.cc:492] T 4d3b3213e1dd426b8efc3fdc733b1842 P a278448771e14471ab4df1e6649485bd: No bootstrap required, opened a new log
I20240501 14:09:44.695187 14393 ts_tablet_manager.cc:1367] T 4d3b3213e1dd426b8efc3fdc733b1842 P a278448771e14471ab4df1e6649485bd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:44.695534 14393 raft_consensus.cc:348] T 4d3b3213e1dd426b8efc3fdc733b1842 P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } }
I20240501 14:09:44.695613 14393 raft_consensus.cc:374] T 4d3b3213e1dd426b8efc3fdc733b1842 P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:44.695649 14393 raft_consensus.cc:729] T 4d3b3213e1dd426b8efc3fdc733b1842 P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a278448771e14471ab4df1e6649485bd, State: Initialized, Role: FOLLOWER
I20240501 14:09:44.695788 14393 consensus_queue.cc:260] T 4d3b3213e1dd426b8efc3fdc733b1842 P a278448771e14471ab4df1e6649485bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } }
I20240501 14:09:44.696028 14393 ts_tablet_manager.cc:1398] T 4d3b3213e1dd426b8efc3fdc733b1842 P a278448771e14471ab4df1e6649485bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:44.834666 14329 raft_consensus.cc:482] T 4d3b3213e1dd426b8efc3fdc733b1842 P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:44.834844 14329 raft_consensus.cc:504] T 4d3b3213e1dd426b8efc3fdc733b1842 P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } }
I20240501 14:09:44.835278 14329 leader_election.cc:288] T 4d3b3213e1dd426b8efc3fdc733b1842 P a278448771e14471ab4df1e6649485bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129:37933)
I20240501 14:09:44.835834 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3b3213e1dd426b8efc3fdc733b1842" candidate_uuid: "a278448771e14471ab4df1e6649485bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50d514f034204fc38331aa0a4fd74a1b" is_pre_election: true
I20240501 14:09:44.835981 13445 raft_consensus.cc:2454] T 4d3b3213e1dd426b8efc3fdc733b1842 P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a278448771e14471ab4df1e6649485bd in term 0.
I20240501 14:09:44.836325 13644 leader_election.cc:302] T 4d3b3213e1dd426b8efc3fdc733b1842 P a278448771e14471ab4df1e6649485bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 50d514f034204fc38331aa0a4fd74a1b, a278448771e14471ab4df1e6649485bd; no voters:
I20240501 14:09:44.836519 14329 raft_consensus.cc:2781] T 4d3b3213e1dd426b8efc3fdc733b1842 P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:44.836602 14329 raft_consensus.cc:482] T 4d3b3213e1dd426b8efc3fdc733b1842 P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:44.836628 14329 raft_consensus.cc:3037] T 4d3b3213e1dd426b8efc3fdc733b1842 P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:44.837416 14329 raft_consensus.cc:504] T 4d3b3213e1dd426b8efc3fdc733b1842 P a278448771e14471ab4df1e6649485bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } }
I20240501 14:09:44.837661 14329 leader_election.cc:288] T 4d3b3213e1dd426b8efc3fdc733b1842 P a278448771e14471ab4df1e6649485bd [CANDIDATE]: Term 1 election: Requested vote from peers 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129:37933)
I20240501 14:09:44.838027 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3b3213e1dd426b8efc3fdc733b1842" candidate_uuid: "a278448771e14471ab4df1e6649485bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50d514f034204fc38331aa0a4fd74a1b"
I20240501 14:09:44.838155 13445 raft_consensus.cc:3037] T 4d3b3213e1dd426b8efc3fdc733b1842 P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:44.839002 13445 raft_consensus.cc:2454] T 4d3b3213e1dd426b8efc3fdc733b1842 P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a278448771e14471ab4df1e6649485bd in term 1.
I20240501 14:09:44.839304 13644 leader_election.cc:302] T 4d3b3213e1dd426b8efc3fdc733b1842 P a278448771e14471ab4df1e6649485bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 50d514f034204fc38331aa0a4fd74a1b, a278448771e14471ab4df1e6649485bd; no voters:
I20240501 14:09:44.839481 14329 raft_consensus.cc:2781] T 4d3b3213e1dd426b8efc3fdc733b1842 P a278448771e14471ab4df1e6649485bd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:44.839565 14329 raft_consensus.cc:686] T 4d3b3213e1dd426b8efc3fdc733b1842 P a278448771e14471ab4df1e6649485bd [term 1 LEADER]: Becoming Leader. State: Replica: a278448771e14471ab4df1e6649485bd, State: Running, Role: LEADER
I20240501 14:09:44.839708 14329 consensus_queue.cc:237] T 4d3b3213e1dd426b8efc3fdc733b1842 P a278448771e14471ab4df1e6649485bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } }
I20240501 14:09:44.841173 13304 catalog_manager.cc:5484] T 4d3b3213e1dd426b8efc3fdc733b1842 P a278448771e14471ab4df1e6649485bd reported cstate change: term changed from 0 to 1, leader changed from <none> to a278448771e14471ab4df1e6649485bd (127.7.39.131). New cstate: current_term: 1 leader_uuid: "a278448771e14471ab4df1e6649485bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } }
I20240501 14:09:44.927227 14338 raft_consensus.cc:482] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:44.927384 14338 raft_consensus.cc:504] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } }
I20240501 14:09:44.927742 14338 leader_election.cc:288] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a278448771e14471ab4df1e6649485bd (127.7.39.131:46461)
I20240501 14:09:44.928248 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba183686332c4ab5b606bbc5f1b4e787" candidate_uuid: "50d514f034204fc38331aa0a4fd74a1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a278448771e14471ab4df1e6649485bd" is_pre_election: true
I20240501 14:09:44.928400 13708 raft_consensus.cc:2454] T ba183686332c4ab5b606bbc5f1b4e787 P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50d514f034204fc38331aa0a4fd74a1b in term 0.
I20240501 14:09:44.928692 13379 leader_election.cc:302] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 50d514f034204fc38331aa0a4fd74a1b, a278448771e14471ab4df1e6649485bd; no voters:
I20240501 14:09:44.928864 14338 raft_consensus.cc:2781] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:44.928922 14338 raft_consensus.cc:482] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:44.928961 14338 raft_consensus.cc:3037] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:44.929811 14338 raft_consensus.cc:504] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } }
I20240501 14:09:44.930063 14338 leader_election.cc:288] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b [CANDIDATE]: Term 1 election: Requested vote from peers a278448771e14471ab4df1e6649485bd (127.7.39.131:46461)
I20240501 14:09:44.930423 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba183686332c4ab5b606bbc5f1b4e787" candidate_uuid: "50d514f034204fc38331aa0a4fd74a1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a278448771e14471ab4df1e6649485bd"
I20240501 14:09:44.930536 13708 raft_consensus.cc:3037] T ba183686332c4ab5b606bbc5f1b4e787 P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:44.931300 13708 raft_consensus.cc:2454] T ba183686332c4ab5b606bbc5f1b4e787 P a278448771e14471ab4df1e6649485bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50d514f034204fc38331aa0a4fd74a1b in term 1.
I20240501 14:09:44.931610 13379 leader_election.cc:302] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 50d514f034204fc38331aa0a4fd74a1b, a278448771e14471ab4df1e6649485bd; no voters:
I20240501 14:09:44.931768 14338 raft_consensus.cc:2781] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:44.931860 14338 raft_consensus.cc:686] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b [term 1 LEADER]: Becoming Leader. State: Replica: 50d514f034204fc38331aa0a4fd74a1b, State: Running, Role: LEADER
I20240501 14:09:44.932003 14338 consensus_queue.cc:237] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } }
I20240501 14:09:44.933521 13304 catalog_manager.cc:5484] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129). New cstate: current_term: 1 leader_uuid: "50d514f034204fc38331aa0a4fd74a1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } }
I20240501 14:09:45.026146 14324 raft_consensus.cc:482] T 0dc94f932e9a4319988e700aff042f22 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:45.026314 14324 raft_consensus.cc:504] T 0dc94f932e9a4319988e700aff042f22 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } }
I20240501 14:09:45.026662 14324 leader_election.cc:288] T 0dc94f932e9a4319988e700aff042f22 P 43736346b1ff48c78ac919c80ba58654 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129:37933)
I20240501 14:09:45.027194 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dc94f932e9a4319988e700aff042f22" candidate_uuid: "43736346b1ff48c78ac919c80ba58654" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50d514f034204fc38331aa0a4fd74a1b" is_pre_election: true
I20240501 14:09:45.027351 13445 raft_consensus.cc:2454] T 0dc94f932e9a4319988e700aff042f22 P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43736346b1ff48c78ac919c80ba58654 in term 0.
I20240501 14:09:45.027679 13513 leader_election.cc:302] T 0dc94f932e9a4319988e700aff042f22 P 43736346b1ff48c78ac919c80ba58654 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 43736346b1ff48c78ac919c80ba58654, 50d514f034204fc38331aa0a4fd74a1b; no voters:
I20240501 14:09:45.027865 14324 raft_consensus.cc:2781] T 0dc94f932e9a4319988e700aff042f22 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:45.027927 14324 raft_consensus.cc:482] T 0dc94f932e9a4319988e700aff042f22 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:45.027976 14324 raft_consensus.cc:3037] T 0dc94f932e9a4319988e700aff042f22 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:45.028841 14324 raft_consensus.cc:504] T 0dc94f932e9a4319988e700aff042f22 P 43736346b1ff48c78ac919c80ba58654 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } }
I20240501 14:09:45.029099 14324 leader_election.cc:288] T 0dc94f932e9a4319988e700aff042f22 P 43736346b1ff48c78ac919c80ba58654 [CANDIDATE]: Term 1 election: Requested vote from peers 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129:37933)
I20240501 14:09:45.029558 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dc94f932e9a4319988e700aff042f22" candidate_uuid: "43736346b1ff48c78ac919c80ba58654" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50d514f034204fc38331aa0a4fd74a1b"
I20240501 14:09:45.029667 13445 raft_consensus.cc:3037] T 0dc94f932e9a4319988e700aff042f22 P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:45.030436 13445 raft_consensus.cc:2454] T 0dc94f932e9a4319988e700aff042f22 P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43736346b1ff48c78ac919c80ba58654 in term 1.
I20240501 14:09:45.030716 13513 leader_election.cc:302] T 0dc94f932e9a4319988e700aff042f22 P 43736346b1ff48c78ac919c80ba58654 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 43736346b1ff48c78ac919c80ba58654, 50d514f034204fc38331aa0a4fd74a1b; no voters:
I20240501 14:09:45.030900 14324 raft_consensus.cc:2781] T 0dc94f932e9a4319988e700aff042f22 P 43736346b1ff48c78ac919c80ba58654 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:45.031035 14324 raft_consensus.cc:686] T 0dc94f932e9a4319988e700aff042f22 P 43736346b1ff48c78ac919c80ba58654 [term 1 LEADER]: Becoming Leader. State: Replica: 43736346b1ff48c78ac919c80ba58654, State: Running, Role: LEADER
I20240501 14:09:45.031221 14324 consensus_queue.cc:237] T 0dc94f932e9a4319988e700aff042f22 P 43736346b1ff48c78ac919c80ba58654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } }
I20240501 14:09:45.032989 13304 catalog_manager.cc:5484] T 0dc94f932e9a4319988e700aff042f22 P 43736346b1ff48c78ac919c80ba58654 reported cstate change: term changed from 0 to 1, leader changed from <none> to 43736346b1ff48c78ac919c80ba58654 (127.7.39.130). New cstate: current_term: 1 leader_uuid: "43736346b1ff48c78ac919c80ba58654" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } }
W20240501 14:09:45.160785 14450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:45.161089 14450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:45.164203 14450 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240501 14:09:45.238759 13445 raft_consensus.cc:1261] T 0dc94f932e9a4319988e700aff042f22 P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Refusing update from remote peer 43736346b1ff48c78ac919c80ba58654: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:45.239238 14324 consensus_queue.cc:1035] T 0dc94f932e9a4319988e700aff042f22 P 43736346b1ff48c78ac919c80ba58654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:45.241161 13446 raft_consensus.cc:1261] T 4d3b3213e1dd426b8efc3fdc733b1842 P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Refusing update from remote peer a278448771e14471ab4df1e6649485bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:45.241626 14329 consensus_queue.cc:1035] T 4d3b3213e1dd426b8efc3fdc733b1842 P a278448771e14471ab4df1e6649485bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:45.241751 13708 raft_consensus.cc:1261] T ba183686332c4ab5b606bbc5f1b4e787 P a278448771e14471ab4df1e6649485bd [term 1 FOLLOWER]: Refusing update from remote peer 50d514f034204fc38331aa0a4fd74a1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:45.243695 14330 consensus_queue.cc:1035] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 88.1871 ms
time per row: 0.0440935 ms
I20240501 14:09:45.295879 13298 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43662:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:45.301332 13687 tablet_service.cc:1467] Processing CreateTablet for tablet bf0b5fd770b8405b9049c52d5d95bc53 (DEFAULT_TABLE table=rebalance_test_table_3 [id=50bc5b382cee4078bfb9cc86c185c1b2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:45.301380 13555 tablet_service.cc:1467] Processing CreateTablet for tablet bf0b5fd770b8405b9049c52d5d95bc53 (DEFAULT_TABLE table=rebalance_test_table_3 [id=50bc5b382cee4078bfb9cc86c185c1b2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:45.301681 13687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf0b5fd770b8405b9049c52d5d95bc53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.301729 13555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf0b5fd770b8405b9049c52d5d95bc53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.303752 14498 tablet_bootstrap.cc:492] T bf0b5fd770b8405b9049c52d5d95bc53 P a278448771e14471ab4df1e6649485bd: Bootstrap starting.
I20240501 14:09:45.303761 14497 tablet_bootstrap.cc:492] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654: Bootstrap starting.
I20240501 14:09:45.301380 13556 tablet_service.cc:1467] Processing CreateTablet for tablet cdb343c0875e4709b9637c241dd4d582 (DEFAULT_TABLE table=rebalance_test_table_3 [id=50bc5b382cee4078bfb9cc86c185c1b2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:45.304098 13556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdb343c0875e4709b9637c241dd4d582. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.304641 14497 tablet_bootstrap.cc:654] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:45.306058 14498 tablet_bootstrap.cc:654] T bf0b5fd770b8405b9049c52d5d95bc53 P a278448771e14471ab4df1e6649485bd: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:45.306228 13555 tablet_service.cc:1467] Processing CreateTablet for tablet cce2260455ac4edbb9bc54b103c50eb3 (DEFAULT_TABLE table=rebalance_test_table_3 [id=50bc5b382cee4078bfb9cc86c185c1b2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:45.306463 13555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cce2260455ac4edbb9bc54b103c50eb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.306475 13425 tablet_service.cc:1467] Processing CreateTablet for tablet cce2260455ac4edbb9bc54b103c50eb3 (DEFAULT_TABLE table=rebalance_test_table_3 [id=50bc5b382cee4078bfb9cc86c185c1b2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:45.306772 13425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cce2260455ac4edbb9bc54b103c50eb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.307320 14498 tablet_bootstrap.cc:492] T bf0b5fd770b8405b9049c52d5d95bc53 P a278448771e14471ab4df1e6649485bd: No bootstrap required, opened a new log
I20240501 14:09:45.307461 14498 ts_tablet_manager.cc:1367] T bf0b5fd770b8405b9049c52d5d95bc53 P a278448771e14471ab4df1e6649485bd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:45.307963 14498 raft_consensus.cc:348] T bf0b5fd770b8405b9049c52d5d95bc53 P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } }
I20240501 14:09:45.308167 14498 raft_consensus.cc:374] T bf0b5fd770b8405b9049c52d5d95bc53 P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:45.308219 14498 raft_consensus.cc:729] T bf0b5fd770b8405b9049c52d5d95bc53 P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a278448771e14471ab4df1e6649485bd, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.308403 14498 consensus_queue.cc:260] T bf0b5fd770b8405b9049c52d5d95bc53 P a278448771e14471ab4df1e6649485bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } }
I20240501 14:09:45.308782 14498 ts_tablet_manager.cc:1398] T bf0b5fd770b8405b9049c52d5d95bc53 P a278448771e14471ab4df1e6649485bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:45.308815 14497 tablet_bootstrap.cc:492] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654: No bootstrap required, opened a new log
I20240501 14:09:45.308933 14497 ts_tablet_manager.cc:1367] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:45.309516 13687 tablet_service.cc:1467] Processing CreateTablet for tablet cdb343c0875e4709b9637c241dd4d582 (DEFAULT_TABLE table=rebalance_test_table_3 [id=50bc5b382cee4078bfb9cc86c185c1b2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:45.309531 14497 raft_consensus.cc:348] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } }
I20240501 14:09:45.309626 14497 raft_consensus.cc:374] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:45.309669 14497 raft_consensus.cc:729] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43736346b1ff48c78ac919c80ba58654, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.309749 13687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdb343c0875e4709b9637c241dd4d582. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.309808 14497 consensus_queue.cc:260] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } }
I20240501 14:09:45.310150 14497 ts_tablet_manager.cc:1398] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:45.310357 14497 tablet_bootstrap.cc:492] T cdb343c0875e4709b9637c241dd4d582 P 43736346b1ff48c78ac919c80ba58654: Bootstrap starting.
I20240501 14:09:45.312136 14498 tablet_bootstrap.cc:492] T cdb343c0875e4709b9637c241dd4d582 P a278448771e14471ab4df1e6649485bd: Bootstrap starting.
I20240501 14:09:45.312273 14501 tablet_bootstrap.cc:492] T cce2260455ac4edbb9bc54b103c50eb3 P 50d514f034204fc38331aa0a4fd74a1b: Bootstrap starting.
I20240501 14:09:45.312158 14497 tablet_bootstrap.cc:654] T cdb343c0875e4709b9637c241dd4d582 P 43736346b1ff48c78ac919c80ba58654: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:45.313010 14498 tablet_bootstrap.cc:654] T cdb343c0875e4709b9637c241dd4d582 P a278448771e14471ab4df1e6649485bd: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:45.313275 14501 tablet_bootstrap.cc:654] T cce2260455ac4edbb9bc54b103c50eb3 P 50d514f034204fc38331aa0a4fd74a1b: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:45.314009 14497 tablet_bootstrap.cc:492] T cdb343c0875e4709b9637c241dd4d582 P 43736346b1ff48c78ac919c80ba58654: No bootstrap required, opened a new log
I20240501 14:09:45.314117 14497 ts_tablet_manager.cc:1367] T cdb343c0875e4709b9637c241dd4d582 P 43736346b1ff48c78ac919c80ba58654: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:45.314065 14498 tablet_bootstrap.cc:492] T cdb343c0875e4709b9637c241dd4d582 P a278448771e14471ab4df1e6649485bd: No bootstrap required, opened a new log
I20240501 14:09:45.314286 14498 ts_tablet_manager.cc:1367] T cdb343c0875e4709b9637c241dd4d582 P a278448771e14471ab4df1e6649485bd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:09:45.314554 14501 tablet_bootstrap.cc:492] T cce2260455ac4edbb9bc54b103c50eb3 P 50d514f034204fc38331aa0a4fd74a1b: No bootstrap required, opened a new log
I20240501 14:09:45.314565 14497 raft_consensus.cc:348] T cdb343c0875e4709b9637c241dd4d582 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } }
I20240501 14:09:45.314649 14501 ts_tablet_manager.cc:1367] T cce2260455ac4edbb9bc54b103c50eb3 P 50d514f034204fc38331aa0a4fd74a1b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:45.314682 14497 raft_consensus.cc:374] T cdb343c0875e4709b9637c241dd4d582 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:45.314731 14497 raft_consensus.cc:729] T cdb343c0875e4709b9637c241dd4d582 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43736346b1ff48c78ac919c80ba58654, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.314865 14497 consensus_queue.cc:260] T cdb343c0875e4709b9637c241dd4d582 P 43736346b1ff48c78ac919c80ba58654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } }
I20240501 14:09:45.315150 14497 ts_tablet_manager.cc:1398] T cdb343c0875e4709b9637c241dd4d582 P 43736346b1ff48c78ac919c80ba58654: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:45.315128 14501 raft_consensus.cc:348] T cce2260455ac4edbb9bc54b103c50eb3 P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } }
I20240501 14:09:45.315230 14501 raft_consensus.cc:374] T cce2260455ac4edbb9bc54b103c50eb3 P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:45.315263 14501 raft_consensus.cc:729] T cce2260455ac4edbb9bc54b103c50eb3 P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50d514f034204fc38331aa0a4fd74a1b, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.315331 14497 tablet_bootstrap.cc:492] T cce2260455ac4edbb9bc54b103c50eb3 P 43736346b1ff48c78ac919c80ba58654: Bootstrap starting.
I20240501 14:09:45.315374 14501 consensus_queue.cc:260] T cce2260455ac4edbb9bc54b103c50eb3 P 50d514f034204fc38331aa0a4fd74a1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } }
I20240501 14:09:45.315675 14501 ts_tablet_manager.cc:1398] T cce2260455ac4edbb9bc54b103c50eb3 P 50d514f034204fc38331aa0a4fd74a1b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:45.315018 14498 raft_consensus.cc:348] T cdb343c0875e4709b9637c241dd4d582 P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } }
I20240501 14:09:45.315840 14498 raft_consensus.cc:374] T cdb343c0875e4709b9637c241dd4d582 P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:45.315872 14498 raft_consensus.cc:729] T cdb343c0875e4709b9637c241dd4d582 P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a278448771e14471ab4df1e6649485bd, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.315976 14498 consensus_queue.cc:260] T cdb343c0875e4709b9637c241dd4d582 P a278448771e14471ab4df1e6649485bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } }
I20240501 14:09:45.316196 14497 tablet_bootstrap.cc:654] T cce2260455ac4edbb9bc54b103c50eb3 P 43736346b1ff48c78ac919c80ba58654: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:45.316833 14498 ts_tablet_manager.cc:1398] T cdb343c0875e4709b9637c241dd4d582 P a278448771e14471ab4df1e6649485bd: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:45.318166 14497 tablet_bootstrap.cc:492] T cce2260455ac4edbb9bc54b103c50eb3 P 43736346b1ff48c78ac919c80ba58654: No bootstrap required, opened a new log
I20240501 14:09:45.318266 14497 ts_tablet_manager.cc:1367] T cce2260455ac4edbb9bc54b103c50eb3 P 43736346b1ff48c78ac919c80ba58654: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:45.318720 14497 raft_consensus.cc:348] T cce2260455ac4edbb9bc54b103c50eb3 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } }
I20240501 14:09:45.318814 14497 raft_consensus.cc:374] T cce2260455ac4edbb9bc54b103c50eb3 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:45.318850 14497 raft_consensus.cc:729] T cce2260455ac4edbb9bc54b103c50eb3 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43736346b1ff48c78ac919c80ba58654, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.318975 14497 consensus_queue.cc:260] T cce2260455ac4edbb9bc54b103c50eb3 P 43736346b1ff48c78ac919c80ba58654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } }
I20240501 14:09:45.319250 14497 ts_tablet_manager.cc:1398] T cce2260455ac4edbb9bc54b103c50eb3 P 43736346b1ff48c78ac919c80ba58654: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:45.348394 14324 raft_consensus.cc:482] T cdb343c0875e4709b9637c241dd4d582 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:45.348544 14324 raft_consensus.cc:504] T cdb343c0875e4709b9637c241dd4d582 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } }
I20240501 14:09:45.348927 14324 leader_election.cc:288] T cdb343c0875e4709b9637c241dd4d582 P 43736346b1ff48c78ac919c80ba58654 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a278448771e14471ab4df1e6649485bd (127.7.39.131:46461)
I20240501 14:09:45.349436 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdb343c0875e4709b9637c241dd4d582" candidate_uuid: "43736346b1ff48c78ac919c80ba58654" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a278448771e14471ab4df1e6649485bd" is_pre_election: true
I20240501 14:09:45.349565 13708 raft_consensus.cc:2454] T cdb343c0875e4709b9637c241dd4d582 P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43736346b1ff48c78ac919c80ba58654 in term 0.
I20240501 14:09:45.349840 13510 leader_election.cc:302] T cdb343c0875e4709b9637c241dd4d582 P 43736346b1ff48c78ac919c80ba58654 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 43736346b1ff48c78ac919c80ba58654, a278448771e14471ab4df1e6649485bd; no voters:
I20240501 14:09:45.350019 14324 raft_consensus.cc:2781] T cdb343c0875e4709b9637c241dd4d582 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:45.350086 14324 raft_consensus.cc:482] T cdb343c0875e4709b9637c241dd4d582 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:45.350124 14324 raft_consensus.cc:3037] T cdb343c0875e4709b9637c241dd4d582 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:45.351135 14324 raft_consensus.cc:504] T cdb343c0875e4709b9637c241dd4d582 P 43736346b1ff48c78ac919c80ba58654 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } }
I20240501 14:09:45.351406 14324 leader_election.cc:288] T cdb343c0875e4709b9637c241dd4d582 P 43736346b1ff48c78ac919c80ba58654 [CANDIDATE]: Term 1 election: Requested vote from peers a278448771e14471ab4df1e6649485bd (127.7.39.131:46461)
I20240501 14:09:45.351766 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdb343c0875e4709b9637c241dd4d582" candidate_uuid: "43736346b1ff48c78ac919c80ba58654" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a278448771e14471ab4df1e6649485bd"
I20240501 14:09:45.351871 13708 raft_consensus.cc:3037] T cdb343c0875e4709b9637c241dd4d582 P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:45.352723 13708 raft_consensus.cc:2454] T cdb343c0875e4709b9637c241dd4d582 P a278448771e14471ab4df1e6649485bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43736346b1ff48c78ac919c80ba58654 in term 1.
I20240501 14:09:45.352996 13510 leader_election.cc:302] T cdb343c0875e4709b9637c241dd4d582 P 43736346b1ff48c78ac919c80ba58654 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 43736346b1ff48c78ac919c80ba58654, a278448771e14471ab4df1e6649485bd; no voters:
I20240501 14:09:45.353158 14324 raft_consensus.cc:2781] T cdb343c0875e4709b9637c241dd4d582 P 43736346b1ff48c78ac919c80ba58654 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:45.353253 14324 raft_consensus.cc:686] T cdb343c0875e4709b9637c241dd4d582 P 43736346b1ff48c78ac919c80ba58654 [term 1 LEADER]: Becoming Leader. State: Replica: 43736346b1ff48c78ac919c80ba58654, State: Running, Role: LEADER
I20240501 14:09:45.353374 14324 consensus_queue.cc:237] T cdb343c0875e4709b9637c241dd4d582 P 43736346b1ff48c78ac919c80ba58654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } }
I20240501 14:09:45.354956 13298 catalog_manager.cc:5484] T cdb343c0875e4709b9637c241dd4d582 P 43736346b1ff48c78ac919c80ba58654 reported cstate change: term changed from 0 to 1, leader changed from <none> to 43736346b1ff48c78ac919c80ba58654 (127.7.39.130). New cstate: current_term: 1 leader_uuid: "43736346b1ff48c78ac919c80ba58654" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } }
I20240501 14:09:45.376967 14341 raft_consensus.cc:482] T cce2260455ac4edbb9bc54b103c50eb3 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:45.377116 14341 raft_consensus.cc:504] T cce2260455ac4edbb9bc54b103c50eb3 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } }
I20240501 14:09:45.377492 14341 leader_election.cc:288] T cce2260455ac4edbb9bc54b103c50eb3 P 43736346b1ff48c78ac919c80ba58654 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129:37933)
I20240501 14:09:45.377976 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cce2260455ac4edbb9bc54b103c50eb3" candidate_uuid: "43736346b1ff48c78ac919c80ba58654" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50d514f034204fc38331aa0a4fd74a1b" is_pre_election: true
I20240501 14:09:45.378124 13446 raft_consensus.cc:2454] T cce2260455ac4edbb9bc54b103c50eb3 P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43736346b1ff48c78ac919c80ba58654 in term 0.
I20240501 14:09:45.378456 13513 leader_election.cc:302] T cce2260455ac4edbb9bc54b103c50eb3 P 43736346b1ff48c78ac919c80ba58654 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 43736346b1ff48c78ac919c80ba58654, 50d514f034204fc38331aa0a4fd74a1b; no voters:
I20240501 14:09:45.378659 14341 raft_consensus.cc:2781] T cce2260455ac4edbb9bc54b103c50eb3 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:45.378723 14341 raft_consensus.cc:482] T cce2260455ac4edbb9bc54b103c50eb3 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:45.378772 14341 raft_consensus.cc:3037] T cce2260455ac4edbb9bc54b103c50eb3 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:45.379662 14341 raft_consensus.cc:504] T cce2260455ac4edbb9bc54b103c50eb3 P 43736346b1ff48c78ac919c80ba58654 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } }
I20240501 14:09:45.379940 14341 leader_election.cc:288] T cce2260455ac4edbb9bc54b103c50eb3 P 43736346b1ff48c78ac919c80ba58654 [CANDIDATE]: Term 1 election: Requested vote from peers 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129:37933)
I20240501 14:09:45.380273 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cce2260455ac4edbb9bc54b103c50eb3" candidate_uuid: "43736346b1ff48c78ac919c80ba58654" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50d514f034204fc38331aa0a4fd74a1b"
I20240501 14:09:45.380380 13446 raft_consensus.cc:3037] T cce2260455ac4edbb9bc54b103c50eb3 P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:45.381119 13446 raft_consensus.cc:2454] T cce2260455ac4edbb9bc54b103c50eb3 P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43736346b1ff48c78ac919c80ba58654 in term 1.
I20240501 14:09:45.381392 13513 leader_election.cc:302] T cce2260455ac4edbb9bc54b103c50eb3 P 43736346b1ff48c78ac919c80ba58654 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 43736346b1ff48c78ac919c80ba58654, 50d514f034204fc38331aa0a4fd74a1b; no voters:
I20240501 14:09:45.381561 14341 raft_consensus.cc:2781] T cce2260455ac4edbb9bc54b103c50eb3 P 43736346b1ff48c78ac919c80ba58654 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:45.381644 14341 raft_consensus.cc:686] T cce2260455ac4edbb9bc54b103c50eb3 P 43736346b1ff48c78ac919c80ba58654 [term 1 LEADER]: Becoming Leader. State: Replica: 43736346b1ff48c78ac919c80ba58654, State: Running, Role: LEADER
I20240501 14:09:45.381815 14341 consensus_queue.cc:237] T cce2260455ac4edbb9bc54b103c50eb3 P 43736346b1ff48c78ac919c80ba58654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } }
I20240501 14:09:45.383672 13298 catalog_manager.cc:5484] T cce2260455ac4edbb9bc54b103c50eb3 P 43736346b1ff48c78ac919c80ba58654 reported cstate change: term changed from 0 to 1, leader changed from <none> to 43736346b1ff48c78ac919c80ba58654 (127.7.39.130). New cstate: current_term: 1 leader_uuid: "43736346b1ff48c78ac919c80ba58654" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } }
I20240501 14:09:45.493594 14324 raft_consensus.cc:482] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:45.493738 14324 raft_consensus.cc:504] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } }
I20240501 14:09:45.494124 14324 leader_election.cc:288] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e26df472efe547f8bc5951dce352471b (127.7.39.134:33035)
W20240501 14:09:45.495608 13511 leader_election.cc:334] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e26df472efe547f8bc5951dce352471b (127.7.39.134:33035): Network error: Client connection negotiation failed: client connection to 127.7.39.134:33035: connect: Connection refused (error 111)
I20240501 14:09:45.495682 13511 leader_election.cc:302] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 43736346b1ff48c78ac919c80ba58654; no voters: e26df472efe547f8bc5951dce352471b
I20240501 14:09:45.495828 14324 raft_consensus.cc:2726] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:09:45.555188 14324 raft_consensus.cc:482] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:45.555331 14324 raft_consensus.cc:504] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } }
I20240501 14:09:45.555684 14324 leader_election.cc:288] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a278448771e14471ab4df1e6649485bd (127.7.39.131:46461)
I20240501 14:09:45.556216 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0b5fd770b8405b9049c52d5d95bc53" candidate_uuid: "43736346b1ff48c78ac919c80ba58654" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a278448771e14471ab4df1e6649485bd" is_pre_election: true
I20240501 14:09:45.556378 13708 raft_consensus.cc:2454] T bf0b5fd770b8405b9049c52d5d95bc53 P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43736346b1ff48c78ac919c80ba58654 in term 0.
I20240501 14:09:45.556677 13510 leader_election.cc:302] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 43736346b1ff48c78ac919c80ba58654, a278448771e14471ab4df1e6649485bd; no voters:
I20240501 14:09:45.556912 14324 raft_consensus.cc:2781] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:45.556963 14324 raft_consensus.cc:482] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:45.557008 14324 raft_consensus.cc:3037] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:45.557834 14324 raft_consensus.cc:504] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } }
I20240501 14:09:45.558116 14324 leader_election.cc:288] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654 [CANDIDATE]: Term 1 election: Requested vote from peers a278448771e14471ab4df1e6649485bd (127.7.39.131:46461)
I20240501 14:09:45.558502 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0b5fd770b8405b9049c52d5d95bc53" candidate_uuid: "43736346b1ff48c78ac919c80ba58654" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a278448771e14471ab4df1e6649485bd"
I20240501 14:09:45.558624 13708 raft_consensus.cc:3037] T bf0b5fd770b8405b9049c52d5d95bc53 P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:45.559659 13708 raft_consensus.cc:2454] T bf0b5fd770b8405b9049c52d5d95bc53 P a278448771e14471ab4df1e6649485bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43736346b1ff48c78ac919c80ba58654 in term 1.
I20240501 14:09:45.559971 13510 leader_election.cc:302] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 43736346b1ff48c78ac919c80ba58654, a278448771e14471ab4df1e6649485bd; no voters:
I20240501 14:09:45.560124 14324 raft_consensus.cc:2781] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:45.560233 14324 raft_consensus.cc:686] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654 [term 1 LEADER]: Becoming Leader. State: Replica: 43736346b1ff48c78ac919c80ba58654, State: Running, Role: LEADER
I20240501 14:09:45.560436 14324 consensus_queue.cc:237] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } }
I20240501 14:09:45.562129 13298 catalog_manager.cc:5484] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654 reported cstate change: term changed from 0 to 1, leader changed from <none> to 43736346b1ff48c78ac919c80ba58654 (127.7.39.130). New cstate: current_term: 1 leader_uuid: "43736346b1ff48c78ac919c80ba58654" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } }
W20240501 14:09:45.701203 14511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:45.701473 14511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:45.704461 14511 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240501 14:09:45.800925 13446 raft_consensus.cc:1261] T cce2260455ac4edbb9bc54b103c50eb3 P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Refusing update from remote peer 43736346b1ff48c78ac919c80ba58654: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:45.801609 13708 raft_consensus.cc:1261] T cdb343c0875e4709b9637c241dd4d582 P a278448771e14471ab4df1e6649485bd [term 1 FOLLOWER]: Refusing update from remote peer 43736346b1ff48c78ac919c80ba58654: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:45.801741 14324 consensus_queue.cc:1035] T cce2260455ac4edbb9bc54b103c50eb3 P 43736346b1ff48c78ac919c80ba58654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:45.801862 13707 raft_consensus.cc:1261] T bf0b5fd770b8405b9049c52d5d95bc53 P a278448771e14471ab4df1e6649485bd [term 1 FOLLOWER]: Refusing update from remote peer 43736346b1ff48c78ac919c80ba58654: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:45.802369 14324 consensus_queue.cc:1035] T cdb343c0875e4709b9637c241dd4d582 P 43736346b1ff48c78ac919c80ba58654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:45.802666 14324 consensus_queue.cc:1035] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 90.4364 ms
time per row: 0.0452182 ms
I20240501 14:09:45.844508 13298 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43662:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:45.850135 13687 tablet_service.cc:1467] Processing CreateTablet for tablet a067a9d6642349c1b99effece908888c (DEFAULT_TABLE table=rebalance_test_table_4 [id=7319ca2094424c85b7008d8839665ed4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:45.850135 13425 tablet_service.cc:1467] Processing CreateTablet for tablet a067a9d6642349c1b99effece908888c (DEFAULT_TABLE table=rebalance_test_table_4 [id=7319ca2094424c85b7008d8839665ed4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:45.850469 13425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a067a9d6642349c1b99effece908888c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.850414 13688 tablet_service.cc:1467] Processing CreateTablet for tablet cde568b4e46a4c5499031d0e18ae51de (DEFAULT_TABLE table=rebalance_test_table_4 [id=7319ca2094424c85b7008d8839665ed4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:45.850469 13687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a067a9d6642349c1b99effece908888c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.850705 13688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cde568b4e46a4c5499031d0e18ae51de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.851080 13424 tablet_service.cc:1467] Processing CreateTablet for tablet f1c30f16a13444cc95132eeb9ea6c96c (DEFAULT_TABLE table=rebalance_test_table_4 [id=7319ca2094424c85b7008d8839665ed4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:45.851315 13424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1c30f16a13444cc95132eeb9ea6c96c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.852725 14557 tablet_bootstrap.cc:492] T cde568b4e46a4c5499031d0e18ae51de P a278448771e14471ab4df1e6649485bd: Bootstrap starting.
I20240501 14:09:45.852928 13426 tablet_service.cc:1467] Processing CreateTablet for tablet cde568b4e46a4c5499031d0e18ae51de (DEFAULT_TABLE table=rebalance_test_table_4 [id=7319ca2094424c85b7008d8839665ed4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:45.853153 13426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cde568b4e46a4c5499031d0e18ae51de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.853989 14557 tablet_bootstrap.cc:654] T cde568b4e46a4c5499031d0e18ae51de P a278448771e14471ab4df1e6649485bd: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:45.854759 14558 tablet_bootstrap.cc:492] T a067a9d6642349c1b99effece908888c P 50d514f034204fc38331aa0a4fd74a1b: Bootstrap starting.
I20240501 14:09:45.855350 13555 tablet_service.cc:1467] Processing CreateTablet for tablet f1c30f16a13444cc95132eeb9ea6c96c (DEFAULT_TABLE table=rebalance_test_table_4 [id=7319ca2094424c85b7008d8839665ed4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:45.855602 13555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1c30f16a13444cc95132eeb9ea6c96c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.855719 14558 tablet_bootstrap.cc:654] T a067a9d6642349c1b99effece908888c P 50d514f034204fc38331aa0a4fd74a1b: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:45.856568 14557 tablet_bootstrap.cc:492] T cde568b4e46a4c5499031d0e18ae51de P a278448771e14471ab4df1e6649485bd: No bootstrap required, opened a new log
I20240501 14:09:45.856715 14557 ts_tablet_manager.cc:1367] T cde568b4e46a4c5499031d0e18ae51de P a278448771e14471ab4df1e6649485bd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:45.856762 14558 tablet_bootstrap.cc:492] T a067a9d6642349c1b99effece908888c P 50d514f034204fc38331aa0a4fd74a1b: No bootstrap required, opened a new log
I20240501 14:09:45.856854 14558 ts_tablet_manager.cc:1367] T a067a9d6642349c1b99effece908888c P 50d514f034204fc38331aa0a4fd74a1b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:45.857223 14558 raft_consensus.cc:348] T a067a9d6642349c1b99effece908888c P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } }
I20240501 14:09:45.857197 14557 raft_consensus.cc:348] T cde568b4e46a4c5499031d0e18ae51de P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } }
I20240501 14:09:45.857317 14557 raft_consensus.cc:374] T cde568b4e46a4c5499031d0e18ae51de P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:45.857318 14558 raft_consensus.cc:374] T a067a9d6642349c1b99effece908888c P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:45.857353 14557 raft_consensus.cc:729] T cde568b4e46a4c5499031d0e18ae51de P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a278448771e14471ab4df1e6649485bd, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.857362 14558 raft_consensus.cc:729] T a067a9d6642349c1b99effece908888c P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50d514f034204fc38331aa0a4fd74a1b, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.857517 14558 consensus_queue.cc:260] T a067a9d6642349c1b99effece908888c P 50d514f034204fc38331aa0a4fd74a1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } }
I20240501 14:09:45.857517 14557 consensus_queue.cc:260] T cde568b4e46a4c5499031d0e18ae51de P a278448771e14471ab4df1e6649485bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } }
I20240501 14:09:45.858106 14561 tablet_bootstrap.cc:492] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654: Bootstrap starting.
I20240501 14:09:45.858328 14558 ts_tablet_manager.cc:1398] T a067a9d6642349c1b99effece908888c P 50d514f034204fc38331aa0a4fd74a1b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:45.858501 14558 tablet_bootstrap.cc:492] T f1c30f16a13444cc95132eeb9ea6c96c P 50d514f034204fc38331aa0a4fd74a1b: Bootstrap starting.
I20240501 14:09:45.858975 14561 tablet_bootstrap.cc:654] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:45.859408 14558 tablet_bootstrap.cc:654] T f1c30f16a13444cc95132eeb9ea6c96c P 50d514f034204fc38331aa0a4fd74a1b: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:45.859529 14557 ts_tablet_manager.cc:1398] T cde568b4e46a4c5499031d0e18ae51de P a278448771e14471ab4df1e6649485bd: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:45.859656 14557 tablet_bootstrap.cc:492] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd: Bootstrap starting.
I20240501 14:09:45.860378 14557 tablet_bootstrap.cc:654] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:45.861017 14558 tablet_bootstrap.cc:492] T f1c30f16a13444cc95132eeb9ea6c96c P 50d514f034204fc38331aa0a4fd74a1b: No bootstrap required, opened a new log
I20240501 14:09:45.861104 14558 ts_tablet_manager.cc:1367] T f1c30f16a13444cc95132eeb9ea6c96c P 50d514f034204fc38331aa0a4fd74a1b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:45.861590 14558 raft_consensus.cc:348] T f1c30f16a13444cc95132eeb9ea6c96c P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } }
I20240501 14:09:45.861698 14558 raft_consensus.cc:374] T f1c30f16a13444cc95132eeb9ea6c96c P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:45.861734 14558 raft_consensus.cc:729] T f1c30f16a13444cc95132eeb9ea6c96c P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50d514f034204fc38331aa0a4fd74a1b, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.861898 14558 consensus_queue.cc:260] T f1c30f16a13444cc95132eeb9ea6c96c P 50d514f034204fc38331aa0a4fd74a1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } }
I20240501 14:09:45.862174 14558 ts_tablet_manager.cc:1398] T f1c30f16a13444cc95132eeb9ea6c96c P 50d514f034204fc38331aa0a4fd74a1b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:45.862425 14558 tablet_bootstrap.cc:492] T cde568b4e46a4c5499031d0e18ae51de P 50d514f034204fc38331aa0a4fd74a1b: Bootstrap starting.
I20240501 14:09:45.862555 14557 tablet_bootstrap.cc:492] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd: No bootstrap required, opened a new log
I20240501 14:09:45.862619 14557 ts_tablet_manager.cc:1367] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:45.862870 14561 tablet_bootstrap.cc:492] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654: No bootstrap required, opened a new log
I20240501 14:09:45.862962 14561 ts_tablet_manager.cc:1367] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:09:45.863095 14557 raft_consensus.cc:348] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } }
I20240501 14:09:45.863188 14557 raft_consensus.cc:374] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:45.863216 14557 raft_consensus.cc:729] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a278448771e14471ab4df1e6649485bd, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.863358 14557 consensus_queue.cc:260] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } }
I20240501 14:09:45.863497 14558 tablet_bootstrap.cc:654] T cde568b4e46a4c5499031d0e18ae51de P 50d514f034204fc38331aa0a4fd74a1b: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:45.863507 14561 raft_consensus.cc:348] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } }
I20240501 14:09:45.863619 14561 raft_consensus.cc:374] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:45.863656 14561 raft_consensus.cc:729] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43736346b1ff48c78ac919c80ba58654, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.863709 14557 ts_tablet_manager.cc:1398] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:45.863795 14561 consensus_queue.cc:260] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } }
I20240501 14:09:45.864097 14561 ts_tablet_manager.cc:1398] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:45.865989 14558 tablet_bootstrap.cc:492] T cde568b4e46a4c5499031d0e18ae51de P 50d514f034204fc38331aa0a4fd74a1b: No bootstrap required, opened a new log
I20240501 14:09:45.866113 14558 ts_tablet_manager.cc:1367] T cde568b4e46a4c5499031d0e18ae51de P 50d514f034204fc38331aa0a4fd74a1b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:45.866531 14558 raft_consensus.cc:348] T cde568b4e46a4c5499031d0e18ae51de P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } }
I20240501 14:09:45.866619 14558 raft_consensus.cc:374] T cde568b4e46a4c5499031d0e18ae51de P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:45.866640 14558 raft_consensus.cc:729] T cde568b4e46a4c5499031d0e18ae51de P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50d514f034204fc38331aa0a4fd74a1b, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.866786 14558 consensus_queue.cc:260] T cde568b4e46a4c5499031d0e18ae51de P 50d514f034204fc38331aa0a4fd74a1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } }
I20240501 14:09:45.867012 14558 ts_tablet_manager.cc:1398] T cde568b4e46a4c5499031d0e18ae51de P 50d514f034204fc38331aa0a4fd74a1b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:45.892796 14543 raft_consensus.cc:482] T cde568b4e46a4c5499031d0e18ae51de P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:45.892931 14543 raft_consensus.cc:504] T cde568b4e46a4c5499031d0e18ae51de P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } }
I20240501 14:09:45.893278 14543 leader_election.cc:288] T cde568b4e46a4c5499031d0e18ae51de P a278448771e14471ab4df1e6649485bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129:37933)
I20240501 14:09:45.893745 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cde568b4e46a4c5499031d0e18ae51de" candidate_uuid: "a278448771e14471ab4df1e6649485bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50d514f034204fc38331aa0a4fd74a1b" is_pre_election: true
I20240501 14:09:45.893913 13445 raft_consensus.cc:2454] T cde568b4e46a4c5499031d0e18ae51de P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a278448771e14471ab4df1e6649485bd in term 0.
I20240501 14:09:45.894265 13644 leader_election.cc:302] T cde568b4e46a4c5499031d0e18ae51de P a278448771e14471ab4df1e6649485bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 50d514f034204fc38331aa0a4fd74a1b, a278448771e14471ab4df1e6649485bd; no voters:
I20240501 14:09:45.894505 14543 raft_consensus.cc:2781] T cde568b4e46a4c5499031d0e18ae51de P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:45.894572 14543 raft_consensus.cc:482] T cde568b4e46a4c5499031d0e18ae51de P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:45.894599 14543 raft_consensus.cc:3037] T cde568b4e46a4c5499031d0e18ae51de P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:45.895448 14543 raft_consensus.cc:504] T cde568b4e46a4c5499031d0e18ae51de P a278448771e14471ab4df1e6649485bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } }
I20240501 14:09:45.895686 14543 leader_election.cc:288] T cde568b4e46a4c5499031d0e18ae51de P a278448771e14471ab4df1e6649485bd [CANDIDATE]: Term 1 election: Requested vote from peers 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129:37933)
I20240501 14:09:45.896029 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cde568b4e46a4c5499031d0e18ae51de" candidate_uuid: "a278448771e14471ab4df1e6649485bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50d514f034204fc38331aa0a4fd74a1b"
I20240501 14:09:45.896131 13445 raft_consensus.cc:3037] T cde568b4e46a4c5499031d0e18ae51de P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:45.896924 13445 raft_consensus.cc:2454] T cde568b4e46a4c5499031d0e18ae51de P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a278448771e14471ab4df1e6649485bd in term 1.
I20240501 14:09:45.897209 13644 leader_election.cc:302] T cde568b4e46a4c5499031d0e18ae51de P a278448771e14471ab4df1e6649485bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 50d514f034204fc38331aa0a4fd74a1b, a278448771e14471ab4df1e6649485bd; no voters:
I20240501 14:09:45.897403 14543 raft_consensus.cc:2781] T cde568b4e46a4c5499031d0e18ae51de P a278448771e14471ab4df1e6649485bd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:45.897487 14543 raft_consensus.cc:686] T cde568b4e46a4c5499031d0e18ae51de P a278448771e14471ab4df1e6649485bd [term 1 LEADER]: Becoming Leader. State: Replica: a278448771e14471ab4df1e6649485bd, State: Running, Role: LEADER
I20240501 14:09:45.897601 14543 consensus_queue.cc:237] T cde568b4e46a4c5499031d0e18ae51de P a278448771e14471ab4df1e6649485bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } }
I20240501 14:09:45.899153 13298 catalog_manager.cc:5484] T cde568b4e46a4c5499031d0e18ae51de P a278448771e14471ab4df1e6649485bd reported cstate change: term changed from 0 to 1, leader changed from <none> to a278448771e14471ab4df1e6649485bd (127.7.39.131). New cstate: current_term: 1 leader_uuid: "a278448771e14471ab4df1e6649485bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } }
I20240501 14:09:45.915843 14543 raft_consensus.cc:482] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:45.915980 14543 raft_consensus.cc:504] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } }
I20240501 14:09:45.916316 14543 leader_election.cc:288] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129:37933)
I20240501 14:09:45.916800 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a067a9d6642349c1b99effece908888c" candidate_uuid: "a278448771e14471ab4df1e6649485bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50d514f034204fc38331aa0a4fd74a1b" is_pre_election: true
I20240501 14:09:45.916945 13445 raft_consensus.cc:2454] T a067a9d6642349c1b99effece908888c P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a278448771e14471ab4df1e6649485bd in term 0.
I20240501 14:09:45.917235 13644 leader_election.cc:302] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 50d514f034204fc38331aa0a4fd74a1b, a278448771e14471ab4df1e6649485bd; no voters:
I20240501 14:09:45.917402 14543 raft_consensus.cc:2781] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:45.917459 14543 raft_consensus.cc:482] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:45.917501 14543 raft_consensus.cc:3037] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:45.918253 14543 raft_consensus.cc:504] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } }
I20240501 14:09:45.918502 14543 leader_election.cc:288] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd [CANDIDATE]: Term 1 election: Requested vote from peers 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129:37933)
I20240501 14:09:45.918853 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a067a9d6642349c1b99effece908888c" candidate_uuid: "a278448771e14471ab4df1e6649485bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50d514f034204fc38331aa0a4fd74a1b"
I20240501 14:09:45.918974 13445 raft_consensus.cc:3037] T a067a9d6642349c1b99effece908888c P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:45.919777 13445 raft_consensus.cc:2454] T a067a9d6642349c1b99effece908888c P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a278448771e14471ab4df1e6649485bd in term 1.
I20240501 14:09:45.920039 13644 leader_election.cc:302] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 50d514f034204fc38331aa0a4fd74a1b, a278448771e14471ab4df1e6649485bd; no voters:
I20240501 14:09:45.920190 14543 raft_consensus.cc:2781] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:45.920269 14543 raft_consensus.cc:686] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd [term 1 LEADER]: Becoming Leader. State: Replica: a278448771e14471ab4df1e6649485bd, State: Running, Role: LEADER
I20240501 14:09:45.920387 14543 consensus_queue.cc:237] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } }
I20240501 14:09:45.921911 13298 catalog_manager.cc:5484] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd reported cstate change: term changed from 0 to 1, leader changed from <none> to a278448771e14471ab4df1e6649485bd (127.7.39.131). New cstate: current_term: 1 leader_uuid: "a278448771e14471ab4df1e6649485bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } }
I20240501 14:09:45.931790 14341 raft_consensus.cc:482] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:45.931911 14341 raft_consensus.cc:504] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } }
I20240501 14:09:45.932236 14341 leader_election.cc:288] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129:37933)
I20240501 14:09:45.932677 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1c30f16a13444cc95132eeb9ea6c96c" candidate_uuid: "43736346b1ff48c78ac919c80ba58654" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50d514f034204fc38331aa0a4fd74a1b" is_pre_election: true
I20240501 14:09:45.932842 13445 raft_consensus.cc:2454] T f1c30f16a13444cc95132eeb9ea6c96c P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43736346b1ff48c78ac919c80ba58654 in term 0.
I20240501 14:09:45.933127 13513 leader_election.cc:302] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 43736346b1ff48c78ac919c80ba58654, 50d514f034204fc38331aa0a4fd74a1b; no voters:
I20240501 14:09:45.933306 14341 raft_consensus.cc:2781] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:45.933363 14341 raft_consensus.cc:482] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:45.933406 14341 raft_consensus.cc:3037] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:45.934154 14341 raft_consensus.cc:504] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } }
I20240501 14:09:45.934432 14341 leader_election.cc:288] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654 [CANDIDATE]: Term 1 election: Requested vote from peers 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129:37933)
I20240501 14:09:45.934808 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1c30f16a13444cc95132eeb9ea6c96c" candidate_uuid: "43736346b1ff48c78ac919c80ba58654" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50d514f034204fc38331aa0a4fd74a1b"
I20240501 14:09:45.934942 13445 raft_consensus.cc:3037] T f1c30f16a13444cc95132eeb9ea6c96c P 50d514f034204fc38331aa0a4fd74a1b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:45.935765 13445 raft_consensus.cc:2454] T f1c30f16a13444cc95132eeb9ea6c96c P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43736346b1ff48c78ac919c80ba58654 in term 1.
I20240501 14:09:45.936009 13513 leader_election.cc:302] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 43736346b1ff48c78ac919c80ba58654, 50d514f034204fc38331aa0a4fd74a1b; no voters:
I20240501 14:09:45.936163 14341 raft_consensus.cc:2781] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:45.936254 14341 raft_consensus.cc:686] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654 [term 1 LEADER]: Becoming Leader. State: Replica: 43736346b1ff48c78ac919c80ba58654, State: Running, Role: LEADER
I20240501 14:09:45.936386 14341 consensus_queue.cc:237] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } }
I20240501 14:09:45.937790 13298 catalog_manager.cc:5484] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654 reported cstate change: term changed from 0 to 1, leader changed from <none> to 43736346b1ff48c78ac919c80ba58654 (127.7.39.130). New cstate: current_term: 1 leader_uuid: "43736346b1ff48c78ac919c80ba58654" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } }
W20240501 14:09:46.059429 14569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:46.059727 14569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:46.062530 14569 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240501 14:09:46.140182 13445 raft_consensus.cc:1261] T a067a9d6642349c1b99effece908888c P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Refusing update from remote peer a278448771e14471ab4df1e6649485bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:46.140856 14543 consensus_queue.cc:1035] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:46.147506 13445 raft_consensus.cc:1261] T cde568b4e46a4c5499031d0e18ae51de P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Refusing update from remote peer a278448771e14471ab4df1e6649485bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:46.150354 14329 consensus_queue.cc:1035] T cde568b4e46a4c5499031d0e18ae51de P a278448771e14471ab4df1e6649485bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:46.150607 13446 raft_consensus.cc:1261] T f1c30f16a13444cc95132eeb9ea6c96c P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Refusing update from remote peer 43736346b1ff48c78ac919c80ba58654: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:46.150975 14341 consensus_queue.cc:1035] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 83.2607 ms
time per row: 0.0416303 ms
I20240501 14:09:46.187213 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.132:43977
--local_ip_for_outbound_sockets=127.7.39.132
--tserver_master_addrs=127.7.39.190:41789
--webserver_port=40497
--webserver_interface=127.7.39.132
--builtin_ntp_servers=127.7.39.148:33673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 14:09:46.294979 14603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:46.295233 14603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:46.295305 14603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:46.298255 14603 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:09:46.298341 14603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:46.298432 14603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.132
I20240501 14:09:46.302078 14603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:33673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.132:43977
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.7.39.132
--webserver_port=40497
--tserver_master_addrs=127.7.39.190:41789
--never_fsync=true
--heap_profile_path=/tmp/kudu.14603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.132
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:46.302968 14603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:46.304065 14603 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:46.313027 14611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:46.314347 14609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:46.315275 14608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:46.316618 14603 server_base.cc:1034] running on GCE node
I20240501 14:09:46.317003 14603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:46.317589 14603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:46.318742 14603 hybrid_clock.cc:648] HybridClock initialized: now 1714572586318730 us; error 31 us; skew 500 ppm
I20240501 14:09:46.320545 14603 webserver.cc:460] Webserver started at http://127.7.39.132:40497/ using document root <none> and password file <none>
I20240501 14:09:46.321094 14603 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:46.321177 14603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:46.324510 14603 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:09:46.326231 14617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:46.327172 14603 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:46.327353 14603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "2496e4af45ba458eacaacc9d25dcf01a"
format_stamp: "Formatted at 2024-05-01 14:09:38 on dist-test-slave-rnfw"
I20240501 14:09:46.327822 14603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:46.339933 14603 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:46.340608 14603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:46.340775 14603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:46.341298 14603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:46.342182 14603 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:46.342237 14603 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:46.342299 14603 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:46.342329 14603 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:46.351171 14603 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.132:43977
I20240501 14:09:46.351187 14730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.132:43977 every 8 connection(s)
I20240501 14:09:46.352293 14603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 14:09:46.353005 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 14603
I20240501 14:09:46.353427 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.133:42905
--local_ip_for_outbound_sockets=127.7.39.133
--tserver_master_addrs=127.7.39.190:41789
--webserver_port=42977
--webserver_interface=127.7.39.133
--builtin_ntp_servers=127.7.39.148:33673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:09:46.361446 14731 heartbeater.cc:347] Connected to a master server at 127.7.39.190:41789
I20240501 14:09:46.361651 14731 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:46.362057 14731 heartbeater.cc:511] Master 127.7.39.190:41789 requested a full tablet report, sending...
I20240501 14:09:46.363255 13295 ts_manager.cc:194] Re-registered known tserver with Master: 2496e4af45ba458eacaacc9d25dcf01a (127.7.39.132:43977)
I20240501 14:09:46.364069 13295 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.132:41809
W20240501 14:09:46.464949 14734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:46.465170 14734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:46.465214 14734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:46.468147 14734 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:09:46.468212 14734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:46.468277 14734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.133
I20240501 14:09:46.471899 14734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:33673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.133:42905
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.7.39.133
--webserver_port=42977
--tserver_master_addrs=127.7.39.190:41789
--never_fsync=true
--heap_profile_path=/tmp/kudu.14734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.133
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:46.472805 14734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:46.473858 14734 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:46.482828 14739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:46.483840 14742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:46.485158 14740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:46.486339 14734 server_base.cc:1034] running on GCE node
I20240501 14:09:46.486742 14734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:46.487372 14734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:46.488523 14734 hybrid_clock.cc:648] HybridClock initialized: now 1714572586488512 us; error 28 us; skew 500 ppm
I20240501 14:09:46.490561 14734 webserver.cc:460] Webserver started at http://127.7.39.133:42977/ using document root <none> and password file <none>
I20240501 14:09:46.491164 14734 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:46.491248 14734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:46.494689 14734 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:09:46.496548 14748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:46.497476 14734 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:46.497570 14734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "4015e9df09424332bc3d7a9bad781641"
format_stamp: "Formatted at 2024-05-01 14:09:39 on dist-test-slave-rnfw"
I20240501 14:09:46.497970 14734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:46.516121 14734 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:46.516743 14734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:46.516911 14734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:46.517453 14734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:46.518309 14734 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:46.518357 14734 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:46.518402 14734 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:46.518435 14734 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:46.527031 14734 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.133:42905
I20240501 14:09:46.527091 14861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.133:42905 every 8 connection(s)
I20240501 14:09:46.528049 14734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 14:09:46.533409 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 14734
I20240501 14:09:46.533813 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.134:33035
--local_ip_for_outbound_sockets=127.7.39.134
--tserver_master_addrs=127.7.39.190:41789
--webserver_port=36143
--webserver_interface=127.7.39.134
--builtin_ntp_servers=127.7.39.148:33673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:09:46.536506 14862 heartbeater.cc:347] Connected to a master server at 127.7.39.190:41789
I20240501 14:09:46.536710 14862 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:46.537139 14862 heartbeater.cc:511] Master 127.7.39.190:41789 requested a full tablet report, sending...
I20240501 14:09:46.538110 13295 ts_manager.cc:194] Re-registered known tserver with Master: 4015e9df09424332bc3d7a9bad781641 (127.7.39.133:42905)
I20240501 14:09:46.538648 13295 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.133:34975
W20240501 14:09:46.640727 14865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:46.640937 14865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:46.640980 14865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:46.643977 14865 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:09:46.644043 14865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:46.644124 14865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.134
I20240501 14:09:46.647620 14865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:33673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.134:33035
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.7.39.134
--webserver_port=36143
--tserver_master_addrs=127.7.39.190:41789
--never_fsync=true
--heap_profile_path=/tmp/kudu.14865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.134
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:46.648517 14865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:46.649653 14865 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:46.659015 14870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:46.659950 14871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:46.661352 14873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:46.662397 14865 server_base.cc:1034] running on GCE node
I20240501 14:09:46.662802 14865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:46.663411 14865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:46.664568 14865 hybrid_clock.cc:648] HybridClock initialized: now 1714572586664543 us; error 44 us; skew 500 ppm
I20240501 14:09:46.666505 14865 webserver.cc:460] Webserver started at http://127.7.39.134:36143/ using document root <none> and password file <none>
I20240501 14:09:46.667068 14865 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:46.667172 14865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:46.670450 14865 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:09:46.672474 14879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:46.673527 14865 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:46.673622 14865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "e26df472efe547f8bc5951dce352471b"
format_stamp: "Formatted at 2024-05-01 14:09:39 on dist-test-slave-rnfw"
I20240501 14:09:46.674059 14865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:46.691529 14865 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:46.692168 14865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:46.692323 14865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:46.692831 14865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:46.693867 14886 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240501 14:09:46.697149 14865 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240501 14:09:46.697221 14865 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20240501 14:09:46.697278 14865 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240501 14:09:46.698673 14865 ts_tablet_manager.cc:590] Registered 1 tablets
I20240501 14:09:46.698712 14865 ts_tablet_manager.cc:569] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:46.698971 14886 tablet_bootstrap.cc:492] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b: Bootstrap starting.
I20240501 14:09:46.709100 14865 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.134:33035
I20240501 14:09:46.709400 14993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.134:33035 every 8 connection(s)
I20240501 14:09:46.710270 14865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 14:09:46.713786 14886 log.cc:824] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:46.715961 14886 tablet_bootstrap.cc:492] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 14:09:46.716778 14886 tablet_bootstrap.cc:492] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b: Bootstrap complete.
I20240501 14:09:46.717906 14886 ts_tablet_manager.cc:1367] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.007s
I20240501 14:09:46.718976 14994 heartbeater.cc:347] Connected to a master server at 127.7.39.190:41789
I20240501 14:09:46.719184 14994 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:46.719583 14994 heartbeater.cc:511] Master 127.7.39.190:41789 requested a full tablet report, sending...
I20240501 14:09:46.720296 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 14865
I20240501 14:09:46.720794 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.135:37645
--local_ip_for_outbound_sockets=127.7.39.135
--tserver_master_addrs=127.7.39.190:41789
--webserver_port=35767
--webserver_interface=127.7.39.135
--builtin_ntp_servers=127.7.39.148:33673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:09:46.720738 14886 raft_consensus.cc:348] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } }
I20240501 14:09:46.721361 14886 raft_consensus.cc:729] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e26df472efe547f8bc5951dce352471b, State: Initialized, Role: FOLLOWER
I20240501 14:09:46.721850 14886 consensus_queue.cc:260] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } }
I20240501 14:09:46.722805 14886 ts_tablet_manager.cc:1398] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240501 14:09:46.723534 13295 ts_manager.cc:194] Re-registered known tserver with Master: e26df472efe547f8bc5951dce352471b (127.7.39.134:33035)
I20240501 14:09:46.724315 13295 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.134:46035
I20240501 14:09:46.726792 14994 heartbeater.cc:503] Master 127.7.39.190:41789 was elected leader, sending a full tablet report...
W20240501 14:09:46.834425 14999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:46.834658 14999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:46.834728 14999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:46.837718 14999 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:09:46.837803 14999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:46.837898 14999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.135
I20240501 14:09:46.841491 14999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:33673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.135:37645
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.7.39.135
--webserver_port=35767
--tserver_master_addrs=127.7.39.190:41789
--never_fsync=true
--heap_profile_path=/tmp/kudu.14999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.135
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:46.842407 14999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:46.843586 14999 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:46.852977 15008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:46.853956 15005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:46.855332 15006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:46.856458 14999 server_base.cc:1034] running on GCE node
I20240501 14:09:46.856897 14999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:46.857517 14999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:46.858738 14999 hybrid_clock.cc:648] HybridClock initialized: now 1714572586858704 us; error 40 us; skew 500 ppm
I20240501 14:09:46.860926 14999 webserver.cc:460] Webserver started at http://127.7.39.135:35767/ using document root <none> and password file <none>
I20240501 14:09:46.861515 14999 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:46.861595 14999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:46.864928 14999 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20240501 14:09:46.866719 15014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:46.867712 14999 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240501 14:09:46.867817 14999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "39bde7c058154f8aa40085db77f8c6e5"
format_stamp: "Formatted at 2024-05-01 14:09:39 on dist-test-slave-rnfw"
I20240501 14:09:46.868265 14999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:46.887138 14999 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:46.887794 14999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:46.887959 14999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:46.888537 14999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:46.889448 14999 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:46.889506 14999 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:46.889564 14999 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:46.889592 14999 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:46.898454 14999 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.135:37645
I20240501 14:09:46.898483 15127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.135:37645 every 8 connection(s)
I20240501 14:09:46.899513 14999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 14:09:46.908396 15128 heartbeater.cc:347] Connected to a master server at 127.7.39.190:41789
I20240501 14:09:46.908669 15128 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:46.909008 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 14999
I20240501 14:09:46.909168 7326 test_util.cc:268] Using random seed: 167509487
I20240501 14:09:46.909204 15128 heartbeater.cc:511] Master 127.7.39.190:41789 requested a full tablet report, sending...
I20240501 14:09:46.911108 13295 ts_manager.cc:194] Re-registered known tserver with Master: 39bde7c058154f8aa40085db77f8c6e5 (127.7.39.135:37645)
I20240501 14:09:46.911784 13295 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.135:34963
I20240501 14:09:46.919274 7326 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20240501 14:09:46.919502 7326 test_util.cc:268] Using random seed: 167519820
I20240501 14:09:46.928212 7326 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20240501 14:09:46.928382 7326 test_util.cc:268] Using random seed: 167528700
I20240501 14:09:46.940189 7326 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20240501 14:09:46.940362 7326 test_util.cc:268] Using random seed: 167540681
I20240501 14:09:46.952313 7326 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20240501 14:09:46.952500 7326 test_util.cc:268] Using random seed: 167552817
W20240501 14:09:46.961342 14995 tablet.cc:2369] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:09:46.964484 7326 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
I20240501 14:09:47.366365 14731 heartbeater.cc:503] Master 127.7.39.190:41789 was elected leader, sending a full tablet report...
I20240501 14:09:47.404651 14665 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:47.410682 13555 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:47.423432 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:47.430122 15062 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:47.437943 14928 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:47.453572 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:47.488759 13687 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:47.540467 14862 heartbeater.cc:503] Master 127.7.39.190:41789 was elected leader, sending a full tablet report...
I20240501 14:09:47.698292 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:47.708290 14665 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:47.714112 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:47.714213 14928 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:47.719271 13555 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:47.719806 15062 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:47.774785 15232 raft_consensus.cc:482] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:47.774904 15232 raft_consensus.cc:504] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } }
I20240501 14:09:47.775209 15232 leader_election.cc:288] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e26df472efe547f8bc5951dce352471b (127.7.39.134:33035)
I20240501 14:09:47.789203 14948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0eb8ac1778a40a0a0e3c6b113ce5741" candidate_uuid: "43736346b1ff48c78ac919c80ba58654" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e26df472efe547f8bc5951dce352471b" is_pre_election: true
I20240501 14:09:47.789537 14948 raft_consensus.cc:2396] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 43736346b1ff48c78ac919c80ba58654 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240501 14:09:47.790244 13511 leader_election.cc:302] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 43736346b1ff48c78ac919c80ba58654; no voters: e26df472efe547f8bc5951dce352471b
I20240501 14:09:47.790596 15232 raft_consensus.cc:2726] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:09:47.800048 13687 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:47.862582 15305 raft_consensus.cc:482] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:47.862915 15305 raft_consensus.cc:504] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } }
I20240501 14:09:47.877815 15305 leader_election.cc:288] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 43736346b1ff48c78ac919c80ba58654 (127.7.39.130:43311)
I20240501 14:09:47.880267 13577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0eb8ac1778a40a0a0e3c6b113ce5741" candidate_uuid: "e26df472efe547f8bc5951dce352471b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "43736346b1ff48c78ac919c80ba58654" is_pre_election: true
I20240501 14:09:47.880440 13577 raft_consensus.cc:2454] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e26df472efe547f8bc5951dce352471b in term 1.
I20240501 14:09:47.880800 14883 leader_election.cc:302] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 43736346b1ff48c78ac919c80ba58654, e26df472efe547f8bc5951dce352471b; no voters:
I20240501 14:09:47.881100 15305 raft_consensus.cc:2781] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:09:47.881189 15305 raft_consensus.cc:482] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:47.881242 15305 raft_consensus.cc:3037] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:47.883963 15305 raft_consensus.cc:504] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } }
I20240501 14:09:47.885156 13577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0eb8ac1778a40a0a0e3c6b113ce5741" candidate_uuid: "e26df472efe547f8bc5951dce352471b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "43736346b1ff48c78ac919c80ba58654"
I20240501 14:09:47.885267 13577 raft_consensus.cc:3037] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:47.889767 15305 leader_election.cc:288] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b [CANDIDATE]: Term 2 election: Requested vote from peers 43736346b1ff48c78ac919c80ba58654 (127.7.39.130:43311)
I20240501 14:09:47.904124 13577 raft_consensus.cc:2454] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e26df472efe547f8bc5951dce352471b in term 2.
I20240501 14:09:47.904486 14883 leader_election.cc:302] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 43736346b1ff48c78ac919c80ba58654, e26df472efe547f8bc5951dce352471b; no voters:
I20240501 14:09:47.905206 15305 raft_consensus.cc:2781] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:47.905421 15305 raft_consensus.cc:686] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b [term 2 LEADER]: Becoming Leader. State: Replica: e26df472efe547f8bc5951dce352471b, State: Running, Role: LEADER
I20240501 14:09:47.905784 15305 consensus_queue.cc:237] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } }
I20240501 14:09:47.908742 13301 catalog_manager.cc:5484] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "e26df472efe547f8bc5951dce352471b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } }
I20240501 14:09:47.914026 15128 heartbeater.cc:503] Master 127.7.39.190:41789 was elected leader, sending a full tablet report...
I20240501 14:09:47.947711 13708 consensus_queue.cc:237] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } }
I20240501 14:09:47.950660 13445 raft_consensus.cc:1261] T a067a9d6642349c1b99effece908888c P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Refusing update from remote peer a278448771e14471ab4df1e6649485bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 14:09:47.952363 14601 consensus_queue.cc:1035] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 14:09:47.954175 14601 raft_consensus.cc:2932] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, VOTER 4015e9df09424332bc3d7a9bad781641 (127.7.39.133) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } }
I20240501 14:09:47.955945 13444 raft_consensus.cc:2932] T a067a9d6642349c1b99effece908888c P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, VOTER 4015e9df09424332bc3d7a9bad781641 (127.7.39.133) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } }
I20240501 14:09:47.962646 13299 catalog_manager.cc:5484] T a067a9d6642349c1b99effece908888c P 50d514f034204fc38331aa0a4fd74a1b reported cstate change: config changed from index -1 to 27, VOTER 4015e9df09424332bc3d7a9bad781641 (127.7.39.133) added. New cstate: current_term: 1 leader_uuid: "a278448771e14471ab4df1e6649485bd" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } }
W20240501 14:09:47.979952 13642 consensus_peers.cc:485] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd -> Peer 4015e9df09424332bc3d7a9bad781641 (127.7.39.133:42905): Couldn't send request to peer 4015e9df09424332bc3d7a9bad781641. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a067a9d6642349c1b99effece908888c. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:48.035485 13577 consensus_queue.cc:237] T cdb343c0875e4709b9637c241dd4d582 P 43736346b1ff48c78ac919c80ba58654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } }
I20240501 14:09:48.038193 13707 raft_consensus.cc:1261] T cdb343c0875e4709b9637c241dd4d582 P a278448771e14471ab4df1e6649485bd [term 1 FOLLOWER]: Refusing update from remote peer 43736346b1ff48c78ac919c80ba58654: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 14:09:48.038646 15180 consensus_queue.cc:1035] T cdb343c0875e4709b9637c241dd4d582 P 43736346b1ff48c78ac919c80ba58654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
W20240501 14:09:48.042696 13511 consensus_peers.cc:485] T cdb343c0875e4709b9637c241dd4d582 P 43736346b1ff48c78ac919c80ba58654 -> Peer 39bde7c058154f8aa40085db77f8c6e5 (127.7.39.135:37645): Couldn't send request to peer 39bde7c058154f8aa40085db77f8c6e5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cdb343c0875e4709b9637c241dd4d582. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:48.043001 15232 raft_consensus.cc:2932] T cdb343c0875e4709b9637c241dd4d582 P 43736346b1ff48c78ac919c80ba58654 [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index -1 to 26, VOTER 39bde7c058154f8aa40085db77f8c6e5 (127.7.39.135) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } } }
I20240501 14:09:48.047304 13299 catalog_manager.cc:5484] T cdb343c0875e4709b9637c241dd4d582 P 43736346b1ff48c78ac919c80ba58654 reported cstate change: config changed from index -1 to 26, VOTER 39bde7c058154f8aa40085db77f8c6e5 (127.7.39.135) added. New cstate: current_term: 1 leader_uuid: "43736346b1ff48c78ac919c80ba58654" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } } }
I20240501 14:09:48.047719 13707 raft_consensus.cc:2932] T cdb343c0875e4709b9637c241dd4d582 P a278448771e14471ab4df1e6649485bd [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, VOTER 39bde7c058154f8aa40085db77f8c6e5 (127.7.39.135) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } } }
I20240501 14:09:48.097242 13707 consensus_queue.cc:237] T cde568b4e46a4c5499031d0e18ae51de P a278448771e14471ab4df1e6649485bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 1.22, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "2496e4af45ba458eacaacc9d25dcf01a" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 43977 } }
I20240501 14:09:48.099411 13443 raft_consensus.cc:1261] T cde568b4e46a4c5499031d0e18ae51de P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Refusing update from remote peer a278448771e14471ab4df1e6649485bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 22. Preceding OpId from leader: term: 1 index: 23. (index mismatch)
I20240501 14:09:48.100047 15203 consensus_queue.cc:1035] T cde568b4e46a4c5499031d0e18ae51de P a278448771e14471ab4df1e6649485bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20240501 14:09:48.104112 15203 raft_consensus.cc:2932] T cde568b4e46a4c5499031d0e18ae51de P a278448771e14471ab4df1e6649485bd [term 1 LEADER]: Committing config change with OpId 1.23: config changed from index -1 to 23, VOTER 2496e4af45ba458eacaacc9d25dcf01a (127.7.39.132) added. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "2496e4af45ba458eacaacc9d25dcf01a" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 43977 } } }
W20240501 14:09:48.105628 13643 consensus_peers.cc:485] T cde568b4e46a4c5499031d0e18ae51de P a278448771e14471ab4df1e6649485bd -> Peer 2496e4af45ba458eacaacc9d25dcf01a (127.7.39.132:43977): Couldn't send request to peer 2496e4af45ba458eacaacc9d25dcf01a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cde568b4e46a4c5499031d0e18ae51de. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:48.106101 13443 raft_consensus.cc:2932] T cde568b4e46a4c5499031d0e18ae51de P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Committing config change with OpId 1.23: config changed from index -1 to 23, VOTER 2496e4af45ba458eacaacc9d25dcf01a (127.7.39.132) added. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "2496e4af45ba458eacaacc9d25dcf01a" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 43977 } } }
I20240501 14:09:48.106838 13299 catalog_manager.cc:5484] T cde568b4e46a4c5499031d0e18ae51de P a278448771e14471ab4df1e6649485bd reported cstate change: config changed from index -1 to 23, VOTER 2496e4af45ba458eacaacc9d25dcf01a (127.7.39.132) added. New cstate: current_term: 1 leader_uuid: "a278448771e14471ab4df1e6649485bd" committed_config { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "2496e4af45ba458eacaacc9d25dcf01a" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 43977 } } }
I20240501 14:09:48.161835 13576 consensus_queue.cc:237] T 0dc94f932e9a4319988e700aff042f22 P 43736346b1ff48c78ac919c80ba58654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 57, Committed index: 57, Last appended: 1.57, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } }
W20240501 14:09:48.163857 13511 consensus_peers.cc:485] T 0dc94f932e9a4319988e700aff042f22 P 43736346b1ff48c78ac919c80ba58654 -> Peer e26df472efe547f8bc5951dce352471b (127.7.39.134:33035): Couldn't send request to peer e26df472efe547f8bc5951dce352471b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0dc94f932e9a4319988e700aff042f22. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:48.166827 13446 raft_consensus.cc:1261] T 0dc94f932e9a4319988e700aff042f22 P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Refusing update from remote peer 43736346b1ff48c78ac919c80ba58654: Log matching property violated. Preceding OpId in replica: term: 1 index: 57. Preceding OpId from leader: term: 1 index: 58. (index mismatch)
I20240501 14:09:48.167410 15233 consensus_queue.cc:1035] T 0dc94f932e9a4319988e700aff042f22 P 43736346b1ff48c78ac919c80ba58654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 58, Last known committed idx: 57, Time since last communication: 0.000s
I20240501 14:09:48.169037 15180 raft_consensus.cc:2932] T 0dc94f932e9a4319988e700aff042f22 P 43736346b1ff48c78ac919c80ba58654 [term 1 LEADER]: Committing config change with OpId 1.58: config changed from index -1 to 58, VOTER e26df472efe547f8bc5951dce352471b (127.7.39.134) added. New config: { opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } }
I20240501 14:09:48.170997 13446 raft_consensus.cc:2932] T 0dc94f932e9a4319988e700aff042f22 P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Committing config change with OpId 1.58: config changed from index -1 to 58, VOTER e26df472efe547f8bc5951dce352471b (127.7.39.134) added. New config: { opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } }
I20240501 14:09:48.171939 13301 catalog_manager.cc:5484] T 0dc94f932e9a4319988e700aff042f22 P 43736346b1ff48c78ac919c80ba58654 reported cstate change: config changed from index -1 to 58, VOTER e26df472efe547f8bc5951dce352471b (127.7.39.134) added. New cstate: current_term: 1 leader_uuid: "43736346b1ff48c78ac919c80ba58654" committed_config { opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } }
I20240501 14:09:48.227453 13577 consensus_queue.cc:237] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 1.34, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } }
I20240501 14:09:48.232630 13707 raft_consensus.cc:1261] T bf0b5fd770b8405b9049c52d5d95bc53 P a278448771e14471ab4df1e6649485bd [term 1 FOLLOWER]: Refusing update from remote peer 43736346b1ff48c78ac919c80ba58654: Log matching property violated. Preceding OpId in replica: term: 1 index: 34. Preceding OpId from leader: term: 1 index: 35. (index mismatch)
I20240501 14:09:48.233901 15212 consensus_queue.cc:1035] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240501 14:09:48.236466 15180 raft_consensus.cc:2932] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654 [term 1 LEADER]: Committing config change with OpId 1.35: config changed from index -1 to 35, VOTER 4015e9df09424332bc3d7a9bad781641 (127.7.39.133) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } }
W20240501 14:09:48.238174 13511 consensus_peers.cc:485] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654 -> Peer 4015e9df09424332bc3d7a9bad781641 (127.7.39.133:42905): Couldn't send request to peer 4015e9df09424332bc3d7a9bad781641. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bf0b5fd770b8405b9049c52d5d95bc53. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:48.239593 13301 catalog_manager.cc:5484] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654 reported cstate change: config changed from index -1 to 35, VOTER 4015e9df09424332bc3d7a9bad781641 (127.7.39.133) added. New cstate: current_term: 1 leader_uuid: "43736346b1ff48c78ac919c80ba58654" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } }
I20240501 14:09:48.239674 13707 raft_consensus.cc:2932] T bf0b5fd770b8405b9049c52d5d95bc53 P a278448771e14471ab4df1e6649485bd [term 1 FOLLOWER]: Committing config change with OpId 1.35: config changed from index -1 to 35, VOTER 4015e9df09424332bc3d7a9bad781641 (127.7.39.133) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } }
I20240501 14:09:48.292621 13577 raft_consensus.cc:1261] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654 [term 2 FOLLOWER]: Refusing update from remote peer e26df472efe547f8bc5951dce352471b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 14:09:48.293428 15318 consensus_queue.cc:1035] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b [LEADER]: Connected to new peer: Peer: permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:48.298256 13446 consensus_queue.cc:237] T 3f34112bba8a43a58420148533a6e989 P 50d514f034204fc38331aa0a4fd74a1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 80, Committed index: 80, Last appended: 1.80, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "2496e4af45ba458eacaacc9d25dcf01a" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 43977 } }
I20240501 14:09:48.299656 13576 raft_consensus.cc:1261] T 3f34112bba8a43a58420148533a6e989 P 43736346b1ff48c78ac919c80ba58654 [term 1 FOLLOWER]: Refusing update from remote peer 50d514f034204fc38331aa0a4fd74a1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 80. Preceding OpId from leader: term: 1 index: 81. (index mismatch)
I20240501 14:09:48.300031 14596 consensus_queue.cc:1035] T 3f34112bba8a43a58420148533a6e989 P 50d514f034204fc38331aa0a4fd74a1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 81, Last known committed idx: 80, Time since last communication: 0.000s
I20240501 14:09:48.303862 14596 raft_consensus.cc:2932] T 3f34112bba8a43a58420148533a6e989 P 50d514f034204fc38331aa0a4fd74a1b [term 1 LEADER]: Committing config change with OpId 1.81: config changed from index -1 to 81, VOTER 2496e4af45ba458eacaacc9d25dcf01a (127.7.39.132) added. New config: { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "2496e4af45ba458eacaacc9d25dcf01a" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 43977 } } }
I20240501 14:09:48.304296 13576 raft_consensus.cc:2932] T 3f34112bba8a43a58420148533a6e989 P 43736346b1ff48c78ac919c80ba58654 [term 1 FOLLOWER]: Committing config change with OpId 1.81: config changed from index -1 to 81, VOTER 2496e4af45ba458eacaacc9d25dcf01a (127.7.39.132) added. New config: { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "2496e4af45ba458eacaacc9d25dcf01a" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 43977 } } }
I20240501 14:09:48.306910 13301 catalog_manager.cc:5484] T 3f34112bba8a43a58420148533a6e989 P 50d514f034204fc38331aa0a4fd74a1b reported cstate change: config changed from index -1 to 81, VOTER 2496e4af45ba458eacaacc9d25dcf01a (127.7.39.132) added. New cstate: current_term: 1 leader_uuid: "50d514f034204fc38331aa0a4fd74a1b" committed_config { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "2496e4af45ba458eacaacc9d25dcf01a" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 43977 } } }
W20240501 14:09:48.315644 13381 consensus_peers.cc:485] T 3f34112bba8a43a58420148533a6e989 P 50d514f034204fc38331aa0a4fd74a1b -> Peer 2496e4af45ba458eacaacc9d25dcf01a (127.7.39.132:43977): Couldn't send request to peer 2496e4af45ba458eacaacc9d25dcf01a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3f34112bba8a43a58420148533a6e989. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:48.410784 13576 consensus_queue.cc:237] T e4266d76760c4f84b16991f8aa38a3c3 P 43736346b1ff48c78ac919c80ba58654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 1.44, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } }
W20240501 14:09:48.412247 13511 consensus_peers.cc:485] T e4266d76760c4f84b16991f8aa38a3c3 P 43736346b1ff48c78ac919c80ba58654 -> Peer 39bde7c058154f8aa40085db77f8c6e5 (127.7.39.135:37645): Couldn't send request to peer 39bde7c058154f8aa40085db77f8c6e5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e4266d76760c4f84b16991f8aa38a3c3. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:48.419816 13445 raft_consensus.cc:1261] T e4266d76760c4f84b16991f8aa38a3c3 P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Refusing update from remote peer 43736346b1ff48c78ac919c80ba58654: Log matching property violated. Preceding OpId in replica: term: 1 index: 44. Preceding OpId from leader: term: 1 index: 45. (index mismatch)
I20240501 14:09:48.420805 15232 consensus_queue.cc:1035] T e4266d76760c4f84b16991f8aa38a3c3 P 43736346b1ff48c78ac919c80ba58654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 45, Last known committed idx: 43, Time since last communication: 0.000s
I20240501 14:09:48.424381 15233 raft_consensus.cc:2932] T e4266d76760c4f84b16991f8aa38a3c3 P 43736346b1ff48c78ac919c80ba58654 [term 1 LEADER]: Committing config change with OpId 1.45: config changed from index -1 to 45, VOTER 39bde7c058154f8aa40085db77f8c6e5 (127.7.39.135) added. New config: { opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } } }
I20240501 14:09:48.428833 13299 catalog_manager.cc:5484] T e4266d76760c4f84b16991f8aa38a3c3 P 43736346b1ff48c78ac919c80ba58654 reported cstate change: config changed from index -1 to 45, VOTER 39bde7c058154f8aa40085db77f8c6e5 (127.7.39.135) added. New cstate: current_term: 1 leader_uuid: "43736346b1ff48c78ac919c80ba58654" committed_config { opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } } }
I20240501 14:09:48.430653 13446 raft_consensus.cc:2932] T e4266d76760c4f84b16991f8aa38a3c3 P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Committing config change with OpId 1.45: config changed from index -1 to 45, VOTER 39bde7c058154f8aa40085db77f8c6e5 (127.7.39.135) added. New config: { opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } } }
I20240501 14:09:48.482656 15434 ts_tablet_manager.cc:902] T cdb343c0875e4709b9637c241dd4d582 P 39bde7c058154f8aa40085db77f8c6e5: Initiating tablet copy from peer 43736346b1ff48c78ac919c80ba58654 (127.7.39.130:43311)
I20240501 14:09:48.483631 15434 tablet_copy_client.cc:308] T cdb343c0875e4709b9637c241dd4d582 P 39bde7c058154f8aa40085db77f8c6e5: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.130:43311
I20240501 14:09:48.486681 15439 ts_tablet_manager.cc:902] T a067a9d6642349c1b99effece908888c P 4015e9df09424332bc3d7a9bad781641: Initiating tablet copy from peer a278448771e14471ab4df1e6649485bd (127.7.39.131:46461)
I20240501 14:09:48.494166 13597 tablet_copy_service.cc:140] P 43736346b1ff48c78ac919c80ba58654: Received BeginTabletCopySession request for tablet cdb343c0875e4709b9637c241dd4d582 from peer 39bde7c058154f8aa40085db77f8c6e5 ({username='slave'} at 127.7.39.135:52733)
I20240501 14:09:48.494343 13597 tablet_copy_service.cc:161] P 43736346b1ff48c78ac919c80ba58654: Beginning new tablet copy session on tablet cdb343c0875e4709b9637c241dd4d582 from peer 39bde7c058154f8aa40085db77f8c6e5 at {username='slave'} at 127.7.39.135:52733: session id = 39bde7c058154f8aa40085db77f8c6e5-cdb343c0875e4709b9637c241dd4d582
I20240501 14:09:48.492152 15439 tablet_copy_client.cc:308] T a067a9d6642349c1b99effece908888c P 4015e9df09424332bc3d7a9bad781641: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.131:46461
I20240501 14:09:48.496692 13597 tablet_copy_source_session.cc:205] T cdb343c0875e4709b9637c241dd4d582 P 43736346b1ff48c78ac919c80ba58654: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:48.502638 15434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdb343c0875e4709b9637c241dd4d582. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:48.506412 13728 tablet_copy_service.cc:140] P a278448771e14471ab4df1e6649485bd: Received BeginTabletCopySession request for tablet a067a9d6642349c1b99effece908888c from peer 4015e9df09424332bc3d7a9bad781641 ({username='slave'} at 127.7.39.133:59567)
I20240501 14:09:48.506572 13728 tablet_copy_service.cc:161] P a278448771e14471ab4df1e6649485bd: Beginning new tablet copy session on tablet a067a9d6642349c1b99effece908888c from peer 4015e9df09424332bc3d7a9bad781641 at {username='slave'} at 127.7.39.133:59567: session id = 4015e9df09424332bc3d7a9bad781641-a067a9d6642349c1b99effece908888c
I20240501 14:09:48.508172 13728 tablet_copy_source_session.cc:205] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:48.512727 15439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a067a9d6642349c1b99effece908888c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:48.513777 15434 tablet_copy_client.cc:783] T cdb343c0875e4709b9637c241dd4d582 P 39bde7c058154f8aa40085db77f8c6e5: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:48.514070 15434 tablet_copy_client.cc:647] T cdb343c0875e4709b9637c241dd4d582 P 39bde7c058154f8aa40085db77f8c6e5: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:48.517082 15434 tablet_copy_client.cc:523] T cdb343c0875e4709b9637c241dd4d582 P 39bde7c058154f8aa40085db77f8c6e5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:48.521469 15434 tablet_bootstrap.cc:492] T cdb343c0875e4709b9637c241dd4d582 P 39bde7c058154f8aa40085db77f8c6e5: Bootstrap starting.
I20240501 14:09:48.526072 15439 tablet_copy_client.cc:783] T a067a9d6642349c1b99effece908888c P 4015e9df09424332bc3d7a9bad781641: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:48.526396 15439 tablet_copy_client.cc:647] T a067a9d6642349c1b99effece908888c P 4015e9df09424332bc3d7a9bad781641: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:48.533237 13707 consensus_queue.cc:237] T d948c2d5182b415b8aedb4c7b786247f P a278448771e14471ab4df1e6649485bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 78, Committed index: 78, Last appended: 1.78, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } }
I20240501 14:09:48.535580 15434 log.cc:824] T cdb343c0875e4709b9637c241dd4d582 P 39bde7c058154f8aa40085db77f8c6e5: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:48.536329 13576 raft_consensus.cc:1261] T d948c2d5182b415b8aedb4c7b786247f P 43736346b1ff48c78ac919c80ba58654 [term 1 FOLLOWER]: Refusing update from remote peer a278448771e14471ab4df1e6649485bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 78. Preceding OpId from leader: term: 1 index: 79. (index mismatch)
I20240501 14:09:48.536873 15000 consensus_queue.cc:1035] T d948c2d5182b415b8aedb4c7b786247f P a278448771e14471ab4df1e6649485bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
I20240501 14:09:48.545290 15000 raft_consensus.cc:2932] T d948c2d5182b415b8aedb4c7b786247f P a278448771e14471ab4df1e6649485bd [term 1 LEADER]: Committing config change with OpId 1.79: config changed from index -1 to 79, VOTER e26df472efe547f8bc5951dce352471b (127.7.39.134) added. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } }
I20240501 14:09:48.546986 13577 raft_consensus.cc:2932] T d948c2d5182b415b8aedb4c7b786247f P 43736346b1ff48c78ac919c80ba58654 [term 1 FOLLOWER]: Committing config change with OpId 1.79: config changed from index -1 to 79, VOTER e26df472efe547f8bc5951dce352471b (127.7.39.134) added. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } }
I20240501 14:09:48.548660 13299 catalog_manager.cc:5484] T d948c2d5182b415b8aedb4c7b786247f P a278448771e14471ab4df1e6649485bd reported cstate change: config changed from index -1 to 79, VOTER e26df472efe547f8bc5951dce352471b (127.7.39.134) added. New cstate: current_term: 1 leader_uuid: "a278448771e14471ab4df1e6649485bd" committed_config { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } }
I20240501 14:09:48.594411 13445 consensus_queue.cc:237] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 57, Committed index: 57, Last appended: 1.57, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } }
I20240501 14:09:48.597180 13707 raft_consensus.cc:1261] T ba183686332c4ab5b606bbc5f1b4e787 P a278448771e14471ab4df1e6649485bd [term 1 FOLLOWER]: Refusing update from remote peer 50d514f034204fc38331aa0a4fd74a1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 57. Preceding OpId from leader: term: 1 index: 58. (index mismatch)
I20240501 14:09:48.597545 15220 consensus_queue.cc:1035] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 58, Last known committed idx: 57, Time since last communication: 0.000s
I20240501 14:09:48.603500 15170 raft_consensus.cc:2932] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b [term 1 LEADER]: Committing config change with OpId 1.58: config changed from index -1 to 58, VOTER 4015e9df09424332bc3d7a9bad781641 (127.7.39.133) added. New config: { opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } }
I20240501 14:09:48.604012 13708 raft_consensus.cc:2932] T ba183686332c4ab5b606bbc5f1b4e787 P a278448771e14471ab4df1e6649485bd [term 1 FOLLOWER]: Committing config change with OpId 1.58: config changed from index -1 to 58, VOTER 4015e9df09424332bc3d7a9bad781641 (127.7.39.133) added. New config: { opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } }
I20240501 14:09:48.606196 13299 catalog_manager.cc:5484] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b reported cstate change: config changed from index -1 to 58, VOTER 4015e9df09424332bc3d7a9bad781641 (127.7.39.133) added. New cstate: current_term: 1 leader_uuid: "50d514f034204fc38331aa0a4fd74a1b" committed_config { opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } }
W20240501 14:09:48.606863 13380 consensus_peers.cc:485] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b -> Peer 4015e9df09424332bc3d7a9bad781641 (127.7.39.133:42905): Couldn't send request to peer 4015e9df09424332bc3d7a9bad781641. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ba183686332c4ab5b606bbc5f1b4e787. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:48.623598 15439 tablet_copy_client.cc:523] T a067a9d6642349c1b99effece908888c P 4015e9df09424332bc3d7a9bad781641: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:48.626046 15439 tablet_bootstrap.cc:492] T a067a9d6642349c1b99effece908888c P 4015e9df09424332bc3d7a9bad781641: Bootstrap starting.
I20240501 14:09:48.656805 15439 log.cc:824] T a067a9d6642349c1b99effece908888c P 4015e9df09424332bc3d7a9bad781641: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:48.659318 15479 ts_tablet_manager.cc:902] T 0dc94f932e9a4319988e700aff042f22 P e26df472efe547f8bc5951dce352471b: Initiating tablet copy from peer 43736346b1ff48c78ac919c80ba58654 (127.7.39.130:43311)
I20240501 14:09:48.660180 15479 tablet_copy_client.cc:308] T 0dc94f932e9a4319988e700aff042f22 P e26df472efe547f8bc5951dce352471b: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.130:43311
I20240501 14:09:48.660794 13597 tablet_copy_service.cc:140] P 43736346b1ff48c78ac919c80ba58654: Received BeginTabletCopySession request for tablet 0dc94f932e9a4319988e700aff042f22 from peer e26df472efe547f8bc5951dce352471b ({username='slave'} at 127.7.39.134:45565)
I20240501 14:09:48.660861 15478 ts_tablet_manager.cc:902] T bf0b5fd770b8405b9049c52d5d95bc53 P 4015e9df09424332bc3d7a9bad781641: Initiating tablet copy from peer 43736346b1ff48c78ac919c80ba58654 (127.7.39.130:43311)
I20240501 14:09:48.660904 13597 tablet_copy_service.cc:161] P 43736346b1ff48c78ac919c80ba58654: Beginning new tablet copy session on tablet 0dc94f932e9a4319988e700aff042f22 from peer e26df472efe547f8bc5951dce352471b at {username='slave'} at 127.7.39.134:45565: session id = e26df472efe547f8bc5951dce352471b-0dc94f932e9a4319988e700aff042f22
W20240501 14:09:48.661298 13642 consensus_peers.cc:485] T d948c2d5182b415b8aedb4c7b786247f P a278448771e14471ab4df1e6649485bd -> Peer e26df472efe547f8bc5951dce352471b (127.7.39.134:33035): Couldn't send request to peer e26df472efe547f8bc5951dce352471b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d948c2d5182b415b8aedb4c7b786247f. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:48.661402 15478 tablet_copy_client.cc:308] T bf0b5fd770b8405b9049c52d5d95bc53 P 4015e9df09424332bc3d7a9bad781641: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.130:43311
I20240501 14:09:48.661856 13597 tablet_copy_source_session.cc:205] T 0dc94f932e9a4319988e700aff042f22 P 43736346b1ff48c78ac919c80ba58654: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:48.663792 15434 tablet_bootstrap.cc:492] T cdb343c0875e4709b9637c241dd4d582 P 39bde7c058154f8aa40085db77f8c6e5: Bootstrap replayed 1/1 log segments. Stats: ops{read=52 overwritten=0 applied=51 ignored=0} inserts{seen=682 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 14:09:48.664361 15434 tablet_bootstrap.cc:492] T cdb343c0875e4709b9637c241dd4d582 P 39bde7c058154f8aa40085db77f8c6e5: Bootstrap complete.
I20240501 14:09:48.664770 15434 ts_tablet_manager.cc:1367] T cdb343c0875e4709b9637c241dd4d582 P 39bde7c058154f8aa40085db77f8c6e5: Time spent bootstrapping tablet: real 0.143s user 0.065s sys 0.005s
I20240501 14:09:48.666033 15479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dc94f932e9a4319988e700aff042f22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:48.669654 15479 tablet_copy_client.cc:783] T 0dc94f932e9a4319988e700aff042f22 P e26df472efe547f8bc5951dce352471b: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:48.669919 15479 tablet_copy_client.cc:647] T 0dc94f932e9a4319988e700aff042f22 P e26df472efe547f8bc5951dce352471b: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:48.673705 15434 raft_consensus.cc:348] T cdb343c0875e4709b9637c241dd4d582 P 39bde7c058154f8aa40085db77f8c6e5 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } }
I20240501 14:09:48.675565 15434 raft_consensus.cc:729] T cdb343c0875e4709b9637c241dd4d582 P 39bde7c058154f8aa40085db77f8c6e5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39bde7c058154f8aa40085db77f8c6e5, State: Initialized, Role: FOLLOWER
I20240501 14:09:48.675891 15458 ts_tablet_manager.cc:902] T cde568b4e46a4c5499031d0e18ae51de P 2496e4af45ba458eacaacc9d25dcf01a: Initiating tablet copy from peer a278448771e14471ab4df1e6649485bd (127.7.39.131:46461)
I20240501 14:09:48.676770 15434 consensus_queue.cc:260] T cdb343c0875e4709b9637c241dd4d582 P 39bde7c058154f8aa40085db77f8c6e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 51, Last appended: 1.52, Last appended by leader: 52, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } }
I20240501 14:09:48.677186 15479 tablet_copy_client.cc:523] T 0dc94f932e9a4319988e700aff042f22 P e26df472efe547f8bc5951dce352471b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:48.677955 15434 ts_tablet_manager.cc:1398] T cdb343c0875e4709b9637c241dd4d582 P 39bde7c058154f8aa40085db77f8c6e5: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20240501 14:09:48.678902 15479 tablet_bootstrap.cc:492] T 0dc94f932e9a4319988e700aff042f22 P e26df472efe547f8bc5951dce352471b: Bootstrap starting.
I20240501 14:09:48.682508 13596 tablet_copy_service.cc:342] P 43736346b1ff48c78ac919c80ba58654: Request end of tablet copy session 39bde7c058154f8aa40085db77f8c6e5-cdb343c0875e4709b9637c241dd4d582 received from {username='slave'} at 127.7.39.135:52733
I20240501 14:09:48.682603 13596 tablet_copy_service.cc:433] P 43736346b1ff48c78ac919c80ba58654: ending tablet copy session 39bde7c058154f8aa40085db77f8c6e5-cdb343c0875e4709b9637c241dd4d582 on tablet cdb343c0875e4709b9637c241dd4d582 with peer 39bde7c058154f8aa40085db77f8c6e5
I20240501 14:09:48.685377 13597 tablet_copy_service.cc:140] P 43736346b1ff48c78ac919c80ba58654: Received BeginTabletCopySession request for tablet bf0b5fd770b8405b9049c52d5d95bc53 from peer 4015e9df09424332bc3d7a9bad781641 ({username='slave'} at 127.7.39.133:35865)
I20240501 14:09:48.685472 13597 tablet_copy_service.cc:161] P 43736346b1ff48c78ac919c80ba58654: Beginning new tablet copy session on tablet bf0b5fd770b8405b9049c52d5d95bc53 from peer 4015e9df09424332bc3d7a9bad781641 at {username='slave'} at 127.7.39.133:35865: session id = 4015e9df09424332bc3d7a9bad781641-bf0b5fd770b8405b9049c52d5d95bc53
I20240501 14:09:48.686385 13597 tablet_copy_source_session.cc:205] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:48.690320 15478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf0b5fd770b8405b9049c52d5d95bc53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:48.691830 15458 tablet_copy_client.cc:308] T cde568b4e46a4c5499031d0e18ae51de P 2496e4af45ba458eacaacc9d25dcf01a: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.131:46461
I20240501 14:09:48.692744 15478 tablet_copy_client.cc:783] T bf0b5fd770b8405b9049c52d5d95bc53 P 4015e9df09424332bc3d7a9bad781641: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:48.692891 15478 tablet_copy_client.cc:647] T bf0b5fd770b8405b9049c52d5d95bc53 P 4015e9df09424332bc3d7a9bad781641: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:48.714350 15478 tablet_copy_client.cc:523] T bf0b5fd770b8405b9049c52d5d95bc53 P 4015e9df09424332bc3d7a9bad781641: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:48.716847 15478 tablet_bootstrap.cc:492] T bf0b5fd770b8405b9049c52d5d95bc53 P 4015e9df09424332bc3d7a9bad781641: Bootstrap starting.
I20240501 14:09:48.732321 13728 tablet_copy_service.cc:140] P a278448771e14471ab4df1e6649485bd: Received BeginTabletCopySession request for tablet cde568b4e46a4c5499031d0e18ae51de from peer 2496e4af45ba458eacaacc9d25dcf01a ({username='slave'} at 127.7.39.132:54871)
I20240501 14:09:48.732456 13728 tablet_copy_service.cc:161] P a278448771e14471ab4df1e6649485bd: Beginning new tablet copy session on tablet cde568b4e46a4c5499031d0e18ae51de from peer 2496e4af45ba458eacaacc9d25dcf01a at {username='slave'} at 127.7.39.132:54871: session id = 2496e4af45ba458eacaacc9d25dcf01a-cde568b4e46a4c5499031d0e18ae51de
I20240501 14:09:48.733438 13728 tablet_copy_source_session.cc:205] T cde568b4e46a4c5499031d0e18ae51de P a278448771e14471ab4df1e6649485bd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:48.738481 15458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cde568b4e46a4c5499031d0e18ae51de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:48.748402 15458 tablet_copy_client.cc:783] T cde568b4e46a4c5499031d0e18ae51de P 2496e4af45ba458eacaacc9d25dcf01a: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:48.748706 15458 tablet_copy_client.cc:647] T cde568b4e46a4c5499031d0e18ae51de P 2496e4af45ba458eacaacc9d25dcf01a: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:48.751221 15458 tablet_copy_client.cc:523] T cde568b4e46a4c5499031d0e18ae51de P 2496e4af45ba458eacaacc9d25dcf01a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:48.760629 15458 tablet_bootstrap.cc:492] T cde568b4e46a4c5499031d0e18ae51de P 2496e4af45ba458eacaacc9d25dcf01a: Bootstrap starting.
I20240501 14:09:48.771692 15505 ts_tablet_manager.cc:902] T 3f34112bba8a43a58420148533a6e989 P 2496e4af45ba458eacaacc9d25dcf01a: Initiating tablet copy from peer 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129:37933)
I20240501 14:09:48.776293 15505 tablet_copy_client.cc:308] T 3f34112bba8a43a58420148533a6e989 P 2496e4af45ba458eacaacc9d25dcf01a: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.129:37933
I20240501 14:09:48.782191 13466 tablet_copy_service.cc:140] P 50d514f034204fc38331aa0a4fd74a1b: Received BeginTabletCopySession request for tablet 3f34112bba8a43a58420148533a6e989 from peer 2496e4af45ba458eacaacc9d25dcf01a ({username='slave'} at 127.7.39.132:51457)
I20240501 14:09:48.782346 13466 tablet_copy_service.cc:161] P 50d514f034204fc38331aa0a4fd74a1b: Beginning new tablet copy session on tablet 3f34112bba8a43a58420148533a6e989 from peer 2496e4af45ba458eacaacc9d25dcf01a at {username='slave'} at 127.7.39.132:51457: session id = 2496e4af45ba458eacaacc9d25dcf01a-3f34112bba8a43a58420148533a6e989
I20240501 14:09:48.783937 13466 tablet_copy_source_session.cc:205] T 3f34112bba8a43a58420148533a6e989 P 50d514f034204fc38331aa0a4fd74a1b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:48.786623 15505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f34112bba8a43a58420148533a6e989. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:48.791925 15458 log.cc:824] T cde568b4e46a4c5499031d0e18ae51de P 2496e4af45ba458eacaacc9d25dcf01a: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:48.793861 15505 tablet_copy_client.cc:783] T 3f34112bba8a43a58420148533a6e989 P 2496e4af45ba458eacaacc9d25dcf01a: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:48.794034 15505 tablet_copy_client.cc:647] T 3f34112bba8a43a58420148533a6e989 P 2496e4af45ba458eacaacc9d25dcf01a: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:48.802237 15505 tablet_copy_client.cc:523] T 3f34112bba8a43a58420148533a6e989 P 2496e4af45ba458eacaacc9d25dcf01a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:48.803869 15505 tablet_bootstrap.cc:492] T 3f34112bba8a43a58420148533a6e989 P 2496e4af45ba458eacaacc9d25dcf01a: Bootstrap starting.
I20240501 14:09:48.811892 13707 consensus_queue.cc:237] T 4d3b3213e1dd426b8efc3fdc733b1842 P a278448771e14471ab4df1e6649485bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 78, Committed index: 78, Last appended: 1.79, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "2496e4af45ba458eacaacc9d25dcf01a" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 43977 } }
W20240501 14:09:48.813798 13643 consensus_peers.cc:485] T 4d3b3213e1dd426b8efc3fdc733b1842 P a278448771e14471ab4df1e6649485bd -> Peer 2496e4af45ba458eacaacc9d25dcf01a (127.7.39.132:43977): Couldn't send request to peer 2496e4af45ba458eacaacc9d25dcf01a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4d3b3213e1dd426b8efc3fdc733b1842. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:48.814118 15439 tablet_bootstrap.cc:492] T a067a9d6642349c1b99effece908888c P 4015e9df09424332bc3d7a9bad781641: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=714 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:48.816176 13442 raft_consensus.cc:1261] T 4d3b3213e1dd426b8efc3fdc733b1842 P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Refusing update from remote peer a278448771e14471ab4df1e6649485bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 78. Preceding OpId from leader: term: 1 index: 80. (index mismatch)
I20240501 14:09:48.819442 15439 tablet_bootstrap.cc:492] T a067a9d6642349c1b99effece908888c P 4015e9df09424332bc3d7a9bad781641: Bootstrap complete.
I20240501 14:09:48.819842 15439 ts_tablet_manager.cc:1367] T a067a9d6642349c1b99effece908888c P 4015e9df09424332bc3d7a9bad781641: Time spent bootstrapping tablet: real 0.194s user 0.057s sys 0.000s
I20240501 14:09:48.823326 15203 consensus_queue.cc:1035] T 4d3b3213e1dd426b8efc3fdc733b1842 P a278448771e14471ab4df1e6649485bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 80, Last known committed idx: 78, Time since last communication: 0.000s
I20240501 14:09:48.830695 15439 raft_consensus.cc:348] T a067a9d6642349c1b99effece908888c P 4015e9df09424332bc3d7a9bad781641 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } }
I20240501 14:09:48.831514 13445 raft_consensus.cc:1203] T 4d3b3213e1dd426b8efc3fdc733b1842 P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.78->[1.79-1.80] Dedup: 1.79->[1.80-1.80]
I20240501 14:09:48.833622 15203 raft_consensus.cc:2932] T 4d3b3213e1dd426b8efc3fdc733b1842 P a278448771e14471ab4df1e6649485bd [term 1 LEADER]: Committing config change with OpId 1.80: config changed from index -1 to 80, VOTER 2496e4af45ba458eacaacc9d25dcf01a (127.7.39.132) added. New config: { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "2496e4af45ba458eacaacc9d25dcf01a" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 43977 } } }
I20240501 14:09:48.835160 15439 raft_consensus.cc:729] T a067a9d6642349c1b99effece908888c P 4015e9df09424332bc3d7a9bad781641 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4015e9df09424332bc3d7a9bad781641, State: Initialized, Role: FOLLOWER
I20240501 14:09:48.835942 15439 consensus_queue.cc:260] T a067a9d6642349c1b99effece908888c P 4015e9df09424332bc3d7a9bad781641 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } }
I20240501 14:09:48.837080 15439 ts_tablet_manager.cc:1398] T a067a9d6642349c1b99effece908888c P 4015e9df09424332bc3d7a9bad781641: Time spent starting tablet: real 0.017s user 0.004s sys 0.000s
I20240501 14:09:48.838996 13301 catalog_manager.cc:5484] T 4d3b3213e1dd426b8efc3fdc733b1842 P a278448771e14471ab4df1e6649485bd reported cstate change: config changed from index -1 to 80, VOTER 2496e4af45ba458eacaacc9d25dcf01a (127.7.39.132) added. New cstate: current_term: 1 leader_uuid: "a278448771e14471ab4df1e6649485bd" committed_config { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "2496e4af45ba458eacaacc9d25dcf01a" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 43977 } } }
I20240501 14:09:48.842295 13445 raft_consensus.cc:2932] T 4d3b3213e1dd426b8efc3fdc733b1842 P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Committing config change with OpId 1.80: config changed from index -1 to 80, VOTER 2496e4af45ba458eacaacc9d25dcf01a (127.7.39.132) added. New config: { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "2496e4af45ba458eacaacc9d25dcf01a" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 43977 } } }
I20240501 14:09:48.842588 13728 tablet_copy_service.cc:342] P a278448771e14471ab4df1e6649485bd: Request end of tablet copy session 4015e9df09424332bc3d7a9bad781641-a067a9d6642349c1b99effece908888c received from {username='slave'} at 127.7.39.133:59567
I20240501 14:09:48.842669 13728 tablet_copy_service.cc:433] P a278448771e14471ab4df1e6649485bd: ending tablet copy session 4015e9df09424332bc3d7a9bad781641-a067a9d6642349c1b99effece908888c on tablet a067a9d6642349c1b99effece908888c with peer 4015e9df09424332bc3d7a9bad781641
I20240501 14:09:48.869005 14816 raft_consensus.cc:1203] T a067a9d6642349c1b99effece908888c P 4015e9df09424332bc3d7a9bad781641 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.26->[1.27-1.35] Dedup: 1.29->[1.30-1.35]
I20240501 14:09:48.878969 15458 tablet_bootstrap.cc:492] T cde568b4e46a4c5499031d0e18ae51de P 2496e4af45ba458eacaacc9d25dcf01a: Bootstrap replayed 1/1 log segments. Stats: ops{read=25 overwritten=0 applied=25 ignored=0} inserts{seen=656 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:48.879578 15458 tablet_bootstrap.cc:492] T cde568b4e46a4c5499031d0e18ae51de P 2496e4af45ba458eacaacc9d25dcf01a: Bootstrap complete.
I20240501 14:09:48.880021 15458 ts_tablet_manager.cc:1367] T cde568b4e46a4c5499031d0e18ae51de P 2496e4af45ba458eacaacc9d25dcf01a: Time spent bootstrapping tablet: real 0.120s user 0.047s sys 0.004s
I20240501 14:09:48.882939 15458 raft_consensus.cc:348] T cde568b4e46a4c5499031d0e18ae51de P 2496e4af45ba458eacaacc9d25dcf01a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "2496e4af45ba458eacaacc9d25dcf01a" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 43977 } }
I20240501 14:09:48.883302 15458 raft_consensus.cc:729] T cde568b4e46a4c5499031d0e18ae51de P 2496e4af45ba458eacaacc9d25dcf01a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2496e4af45ba458eacaacc9d25dcf01a, State: Initialized, Role: FOLLOWER
I20240501 14:09:48.883860 15458 consensus_queue.cc:260] T cde568b4e46a4c5499031d0e18ae51de P 2496e4af45ba458eacaacc9d25dcf01a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "2496e4af45ba458eacaacc9d25dcf01a" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 43977 } }
I20240501 14:09:48.884867 15458 ts_tablet_manager.cc:1398] T cde568b4e46a4c5499031d0e18ae51de P 2496e4af45ba458eacaacc9d25dcf01a: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240501 14:09:48.888237 13728 tablet_copy_service.cc:342] P a278448771e14471ab4df1e6649485bd: Request end of tablet copy session 2496e4af45ba458eacaacc9d25dcf01a-cde568b4e46a4c5499031d0e18ae51de received from {username='slave'} at 127.7.39.132:54871
I20240501 14:09:48.888338 13728 tablet_copy_service.cc:433] P a278448771e14471ab4df1e6649485bd: ending tablet copy session 2496e4af45ba458eacaacc9d25dcf01a-cde568b4e46a4c5499031d0e18ae51de on tablet cde568b4e46a4c5499031d0e18ae51de with peer 2496e4af45ba458eacaacc9d25dcf01a
I20240501 14:09:48.900079 15478 tablet_bootstrap.cc:492] T bf0b5fd770b8405b9049c52d5d95bc53 P 4015e9df09424332bc3d7a9bad781641: Bootstrap replayed 1/1 log segments. Stats: ops{read=52 overwritten=0 applied=52 ignored=0} inserts{seen=737 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:48.900223 15479 tablet_bootstrap.cc:492] T 0dc94f932e9a4319988e700aff042f22 P e26df472efe547f8bc5951dce352471b: Bootstrap replayed 1/1 log segments. Stats: ops{read=80 overwritten=0 applied=79 ignored=0} inserts{seen=751 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 14:09:48.900460 15478 tablet_bootstrap.cc:492] T bf0b5fd770b8405b9049c52d5d95bc53 P 4015e9df09424332bc3d7a9bad781641: Bootstrap complete.
I20240501 14:09:48.900535 15479 tablet_bootstrap.cc:492] T 0dc94f932e9a4319988e700aff042f22 P e26df472efe547f8bc5951dce352471b: Bootstrap complete.
I20240501 14:09:48.900669 15478 ts_tablet_manager.cc:1367] T bf0b5fd770b8405b9049c52d5d95bc53 P 4015e9df09424332bc3d7a9bad781641: Time spent bootstrapping tablet: real 0.184s user 0.062s sys 0.000s
I20240501 14:09:48.900712 15479 ts_tablet_manager.cc:1367] T 0dc94f932e9a4319988e700aff042f22 P e26df472efe547f8bc5951dce352471b: Time spent bootstrapping tablet: real 0.222s user 0.072s sys 0.015s
I20240501 14:09:48.901218 15478 raft_consensus.cc:348] T bf0b5fd770b8405b9049c52d5d95bc53 P 4015e9df09424332bc3d7a9bad781641 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } }
I20240501 14:09:48.901266 15479 raft_consensus.cc:348] T 0dc94f932e9a4319988e700aff042f22 P e26df472efe547f8bc5951dce352471b [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } }
I20240501 14:09:48.901340 15478 raft_consensus.cc:729] T bf0b5fd770b8405b9049c52d5d95bc53 P 4015e9df09424332bc3d7a9bad781641 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4015e9df09424332bc3d7a9bad781641, State: Initialized, Role: FOLLOWER
I20240501 14:09:48.902539 15479 raft_consensus.cc:729] T 0dc94f932e9a4319988e700aff042f22 P e26df472efe547f8bc5951dce352471b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e26df472efe547f8bc5951dce352471b, State: Initialized, Role: FOLLOWER
I20240501 14:09:48.902770 15479 consensus_queue.cc:260] T 0dc94f932e9a4319988e700aff042f22 P e26df472efe547f8bc5951dce352471b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 79, Last appended: 1.80, Last appended by leader: 80, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } }
I20240501 14:09:48.904603 15478 consensus_queue.cc:260] T bf0b5fd770b8405b9049c52d5d95bc53 P 4015e9df09424332bc3d7a9bad781641 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 52, Last appended: 1.52, Last appended by leader: 52, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } }
I20240501 14:09:48.904752 15082 raft_consensus.cc:1203] T cdb343c0875e4709b9637c241dd4d582 P 39bde7c058154f8aa40085db77f8c6e5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.25->[1.26-1.61] Dedup: 1.52->[1.53-1.61]
I20240501 14:09:48.912684 15478 ts_tablet_manager.cc:1398] T bf0b5fd770b8405b9049c52d5d95bc53 P 4015e9df09424332bc3d7a9bad781641: Time spent starting tablet: real 0.012s user 0.003s sys 0.000s
I20240501 14:09:48.913420 13597 tablet_copy_service.cc:342] P 43736346b1ff48c78ac919c80ba58654: Request end of tablet copy session 4015e9df09424332bc3d7a9bad781641-bf0b5fd770b8405b9049c52d5d95bc53 received from {username='slave'} at 127.7.39.133:35865
I20240501 14:09:48.913507 13597 tablet_copy_service.cc:433] P 43736346b1ff48c78ac919c80ba58654: ending tablet copy session 4015e9df09424332bc3d7a9bad781641-bf0b5fd770b8405b9049c52d5d95bc53 on tablet bf0b5fd770b8405b9049c52d5d95bc53 with peer 4015e9df09424332bc3d7a9bad781641
I20240501 14:09:48.917573 15479 ts_tablet_manager.cc:1398] T 0dc94f932e9a4319988e700aff042f22 P e26df472efe547f8bc5951dce352471b: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20240501 14:09:48.924168 13597 tablet_copy_service.cc:342] P 43736346b1ff48c78ac919c80ba58654: Request end of tablet copy session e26df472efe547f8bc5951dce352471b-0dc94f932e9a4319988e700aff042f22 received from {username='slave'} at 127.7.39.134:45565
I20240501 14:09:48.924248 13597 tablet_copy_service.cc:433] P 43736346b1ff48c78ac919c80ba58654: ending tablet copy session e26df472efe547f8bc5951dce352471b-0dc94f932e9a4319988e700aff042f22 on tablet 0dc94f932e9a4319988e700aff042f22 with peer e26df472efe547f8bc5951dce352471b
I20240501 14:09:49.019730 15434 ts_tablet_manager.cc:902] T e4266d76760c4f84b16991f8aa38a3c3 P 39bde7c058154f8aa40085db77f8c6e5: Initiating tablet copy from peer 43736346b1ff48c78ac919c80ba58654 (127.7.39.130:43311)
I20240501 14:09:49.020417 15434 tablet_copy_client.cc:308] T e4266d76760c4f84b16991f8aa38a3c3 P 39bde7c058154f8aa40085db77f8c6e5: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.130:43311
I20240501 14:09:49.021838 13597 tablet_copy_service.cc:140] P 43736346b1ff48c78ac919c80ba58654: Received BeginTabletCopySession request for tablet e4266d76760c4f84b16991f8aa38a3c3 from peer 39bde7c058154f8aa40085db77f8c6e5 ({username='slave'} at 127.7.39.135:52733)
I20240501 14:09:49.021948 13597 tablet_copy_service.cc:161] P 43736346b1ff48c78ac919c80ba58654: Beginning new tablet copy session on tablet e4266d76760c4f84b16991f8aa38a3c3 from peer 39bde7c058154f8aa40085db77f8c6e5 at {username='slave'} at 127.7.39.135:52733: session id = 39bde7c058154f8aa40085db77f8c6e5-e4266d76760c4f84b16991f8aa38a3c3
I20240501 14:09:49.022863 13597 tablet_copy_source_session.cc:205] T e4266d76760c4f84b16991f8aa38a3c3 P 43736346b1ff48c78ac919c80ba58654: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:49.024019 15434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4266d76760c4f84b16991f8aa38a3c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:49.026361 15434 tablet_copy_client.cc:783] T e4266d76760c4f84b16991f8aa38a3c3 P 39bde7c058154f8aa40085db77f8c6e5: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:49.026510 15434 tablet_copy_client.cc:647] T e4266d76760c4f84b16991f8aa38a3c3 P 39bde7c058154f8aa40085db77f8c6e5: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:49.028412 13577 consensus_queue.cc:237] T 8a89cc8e4b2b476ba954d3cc64537031 P 43736346b1ff48c78ac919c80ba58654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 1.95, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } }
W20240501 14:09:49.029939 13511 consensus_peers.cc:485] T 8a89cc8e4b2b476ba954d3cc64537031 P 43736346b1ff48c78ac919c80ba58654 -> Peer 39bde7c058154f8aa40085db77f8c6e5 (127.7.39.135:37645): Couldn't send request to peer 39bde7c058154f8aa40085db77f8c6e5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8a89cc8e4b2b476ba954d3cc64537031. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:49.033345 15434 tablet_copy_client.cc:523] T e4266d76760c4f84b16991f8aa38a3c3 P 39bde7c058154f8aa40085db77f8c6e5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:49.034997 15434 tablet_bootstrap.cc:492] T e4266d76760c4f84b16991f8aa38a3c3 P 39bde7c058154f8aa40085db77f8c6e5: Bootstrap starting.
I20240501 14:09:49.038931 13708 raft_consensus.cc:1261] T 8a89cc8e4b2b476ba954d3cc64537031 P a278448771e14471ab4df1e6649485bd [term 1 FOLLOWER]: Refusing update from remote peer 43736346b1ff48c78ac919c80ba58654: Log matching property violated. Preceding OpId in replica: term: 1 index: 95. Preceding OpId from leader: term: 1 index: 96. (index mismatch)
I20240501 14:09:49.039319 15233 consensus_queue.cc:1035] T 8a89cc8e4b2b476ba954d3cc64537031 P 43736346b1ff48c78ac919c80ba58654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20240501 14:09:49.041927 15180 raft_consensus.cc:2932] T 8a89cc8e4b2b476ba954d3cc64537031 P 43736346b1ff48c78ac919c80ba58654 [term 1 LEADER]: Committing config change with OpId 1.96: config changed from index -1 to 96, VOTER 39bde7c058154f8aa40085db77f8c6e5 (127.7.39.135) added. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } } }
I20240501 14:09:49.042351 13707 raft_consensus.cc:2932] T 8a89cc8e4b2b476ba954d3cc64537031 P a278448771e14471ab4df1e6649485bd [term 1 FOLLOWER]: Committing config change with OpId 1.96: config changed from index -1 to 96, VOTER 39bde7c058154f8aa40085db77f8c6e5 (127.7.39.135) added. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } } }
I20240501 14:09:49.045512 15478 ts_tablet_manager.cc:902] T ba183686332c4ab5b606bbc5f1b4e787 P 4015e9df09424332bc3d7a9bad781641: Initiating tablet copy from peer 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129:37933)
I20240501 14:09:49.045871 13299 catalog_manager.cc:5484] T 8a89cc8e4b2b476ba954d3cc64537031 P 43736346b1ff48c78ac919c80ba58654 reported cstate change: config changed from index -1 to 96, VOTER 39bde7c058154f8aa40085db77f8c6e5 (127.7.39.135) added. New cstate: current_term: 1 leader_uuid: "43736346b1ff48c78ac919c80ba58654" committed_config { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } } }
I20240501 14:09:49.046028 15478 tablet_copy_client.cc:308] T ba183686332c4ab5b606bbc5f1b4e787 P 4015e9df09424332bc3d7a9bad781641: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.129:37933
I20240501 14:09:49.052786 13466 tablet_copy_service.cc:140] P 50d514f034204fc38331aa0a4fd74a1b: Received BeginTabletCopySession request for tablet ba183686332c4ab5b606bbc5f1b4e787 from peer 4015e9df09424332bc3d7a9bad781641 ({username='slave'} at 127.7.39.133:33105)
I20240501 14:09:49.052898 13466 tablet_copy_service.cc:161] P 50d514f034204fc38331aa0a4fd74a1b: Beginning new tablet copy session on tablet ba183686332c4ab5b606bbc5f1b4e787 from peer 4015e9df09424332bc3d7a9bad781641 at {username='slave'} at 127.7.39.133:33105: session id = 4015e9df09424332bc3d7a9bad781641-ba183686332c4ab5b606bbc5f1b4e787
I20240501 14:09:49.053826 13466 tablet_copy_source_session.cc:205] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:49.066640 15478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba183686332c4ab5b606bbc5f1b4e787. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:49.071344 15478 tablet_copy_client.cc:783] T ba183686332c4ab5b606bbc5f1b4e787 P 4015e9df09424332bc3d7a9bad781641: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:49.071568 15478 tablet_copy_client.cc:647] T ba183686332c4ab5b606bbc5f1b4e787 P 4015e9df09424332bc3d7a9bad781641: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:49.074982 15478 tablet_copy_client.cc:523] T ba183686332c4ab5b606bbc5f1b4e787 P 4015e9df09424332bc3d7a9bad781641: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:49.077646 15478 tablet_bootstrap.cc:492] T ba183686332c4ab5b606bbc5f1b4e787 P 4015e9df09424332bc3d7a9bad781641: Bootstrap starting.
I20240501 14:09:49.106319 15479 ts_tablet_manager.cc:902] T d948c2d5182b415b8aedb4c7b786247f P e26df472efe547f8bc5951dce352471b: Initiating tablet copy from peer a278448771e14471ab4df1e6649485bd (127.7.39.131:46461)
I20240501 14:09:49.106784 15479 tablet_copy_client.cc:308] T d948c2d5182b415b8aedb4c7b786247f P e26df472efe547f8bc5951dce352471b: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.131:46461
I20240501 14:09:49.111429 15505 tablet_bootstrap.cc:492] T 3f34112bba8a43a58420148533a6e989 P 2496e4af45ba458eacaacc9d25dcf01a: Bootstrap replayed 1/1 log segments. Stats: ops{read=99 overwritten=0 applied=99 ignored=0} inserts{seen=771 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:49.114693 15505 tablet_bootstrap.cc:492] T 3f34112bba8a43a58420148533a6e989 P 2496e4af45ba458eacaacc9d25dcf01a: Bootstrap complete.
I20240501 14:09:49.114969 15505 ts_tablet_manager.cc:1367] T 3f34112bba8a43a58420148533a6e989 P 2496e4af45ba458eacaacc9d25dcf01a: Time spent bootstrapping tablet: real 0.311s user 0.088s sys 0.013s
I20240501 14:09:49.115666 15505 raft_consensus.cc:348] T 3f34112bba8a43a58420148533a6e989 P 2496e4af45ba458eacaacc9d25dcf01a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "2496e4af45ba458eacaacc9d25dcf01a" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 43977 } }
I20240501 14:09:49.115906 15505 raft_consensus.cc:729] T 3f34112bba8a43a58420148533a6e989 P 2496e4af45ba458eacaacc9d25dcf01a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2496e4af45ba458eacaacc9d25dcf01a, State: Initialized, Role: FOLLOWER
I20240501 14:09:49.116130 15505 consensus_queue.cc:260] T 3f34112bba8a43a58420148533a6e989 P 2496e4af45ba458eacaacc9d25dcf01a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 99, Last appended: 1.99, Last appended by leader: 99, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "2496e4af45ba458eacaacc9d25dcf01a" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 43977 } }
I20240501 14:09:49.117902 14685 raft_consensus.cc:1203] T cde568b4e46a4c5499031d0e18ae51de P 2496e4af45ba458eacaacc9d25dcf01a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.22->[1.23-1.37] Dedup: 1.25->[1.26-1.37]
I20240501 14:09:49.129565 15505 ts_tablet_manager.cc:1398] T 3f34112bba8a43a58420148533a6e989 P 2496e4af45ba458eacaacc9d25dcf01a: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20240501 14:09:49.131445 13466 tablet_copy_service.cc:342] P 50d514f034204fc38331aa0a4fd74a1b: Request end of tablet copy session 2496e4af45ba458eacaacc9d25dcf01a-3f34112bba8a43a58420148533a6e989 received from {username='slave'} at 127.7.39.132:51457
I20240501 14:09:49.131592 13466 tablet_copy_service.cc:433] P 50d514f034204fc38331aa0a4fd74a1b: ending tablet copy session 2496e4af45ba458eacaacc9d25dcf01a-3f34112bba8a43a58420148533a6e989 on tablet 3f34112bba8a43a58420148533a6e989 with peer 2496e4af45ba458eacaacc9d25dcf01a
I20240501 14:09:49.191671 15478 tablet_bootstrap.cc:492] T ba183686332c4ab5b606bbc5f1b4e787 P 4015e9df09424332bc3d7a9bad781641: Bootstrap replayed 1/1 log segments. Stats: ops{read=70 overwritten=0 applied=70 ignored=0} inserts{seen=755 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:49.192132 15478 tablet_bootstrap.cc:492] T ba183686332c4ab5b606bbc5f1b4e787 P 4015e9df09424332bc3d7a9bad781641: Bootstrap complete.
I20240501 14:09:49.192430 15478 ts_tablet_manager.cc:1367] T ba183686332c4ab5b606bbc5f1b4e787 P 4015e9df09424332bc3d7a9bad781641: Time spent bootstrapping tablet: real 0.115s user 0.069s sys 0.006s
I20240501 14:09:49.192993 15478 raft_consensus.cc:348] T ba183686332c4ab5b606bbc5f1b4e787 P 4015e9df09424332bc3d7a9bad781641 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } }
I20240501 14:09:49.193189 15478 raft_consensus.cc:729] T ba183686332c4ab5b606bbc5f1b4e787 P 4015e9df09424332bc3d7a9bad781641 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4015e9df09424332bc3d7a9bad781641, State: Initialized, Role: FOLLOWER
I20240501 14:09:49.193540 15478 consensus_queue.cc:260] T ba183686332c4ab5b606bbc5f1b4e787 P 4015e9df09424332bc3d7a9bad781641 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 70, Last appended: 1.70, Last appended by leader: 70, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } }
I20240501 14:09:49.195979 15434 tablet_bootstrap.cc:492] T e4266d76760c4f84b16991f8aa38a3c3 P 39bde7c058154f8aa40085db77f8c6e5: Bootstrap replayed 1/1 log segments. Stats: ops{read=61 overwritten=0 applied=61 ignored=0} inserts{seen=733 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:49.196389 15434 tablet_bootstrap.cc:492] T e4266d76760c4f84b16991f8aa38a3c3 P 39bde7c058154f8aa40085db77f8c6e5: Bootstrap complete.
I20240501 14:09:49.196666 15434 ts_tablet_manager.cc:1367] T e4266d76760c4f84b16991f8aa38a3c3 P 39bde7c058154f8aa40085db77f8c6e5: Time spent bootstrapping tablet: real 0.162s user 0.071s sys 0.004s
I20240501 14:09:49.196944 15478 ts_tablet_manager.cc:1398] T ba183686332c4ab5b606bbc5f1b4e787 P 4015e9df09424332bc3d7a9bad781641: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240501 14:09:49.197806 15434 raft_consensus.cc:348] T e4266d76760c4f84b16991f8aa38a3c3 P 39bde7c058154f8aa40085db77f8c6e5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } }
I20240501 14:09:49.198016 15434 raft_consensus.cc:729] T e4266d76760c4f84b16991f8aa38a3c3 P 39bde7c058154f8aa40085db77f8c6e5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39bde7c058154f8aa40085db77f8c6e5, State: Initialized, Role: FOLLOWER
I20240501 14:09:49.198271 15434 consensus_queue.cc:260] T e4266d76760c4f84b16991f8aa38a3c3 P 39bde7c058154f8aa40085db77f8c6e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 61, Last appended: 1.61, Last appended by leader: 61, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } }
I20240501 14:09:49.199434 15434 ts_tablet_manager.cc:1398] T e4266d76760c4f84b16991f8aa38a3c3 P 39bde7c058154f8aa40085db77f8c6e5: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:49.200328 13597 tablet_copy_service.cc:342] P 43736346b1ff48c78ac919c80ba58654: Request end of tablet copy session 39bde7c058154f8aa40085db77f8c6e5-e4266d76760c4f84b16991f8aa38a3c3 received from {username='slave'} at 127.7.39.135:52733
I20240501 14:09:49.200430 13597 tablet_copy_service.cc:433] P 43736346b1ff48c78ac919c80ba58654: ending tablet copy session 39bde7c058154f8aa40085db77f8c6e5-e4266d76760c4f84b16991f8aa38a3c3 on tablet e4266d76760c4f84b16991f8aa38a3c3 with peer 39bde7c058154f8aa40085db77f8c6e5
I20240501 14:09:49.200902 13466 tablet_copy_service.cc:342] P 50d514f034204fc38331aa0a4fd74a1b: Request end of tablet copy session 4015e9df09424332bc3d7a9bad781641-ba183686332c4ab5b606bbc5f1b4e787 received from {username='slave'} at 127.7.39.133:33105
I20240501 14:09:49.200990 13466 tablet_copy_service.cc:433] P 50d514f034204fc38331aa0a4fd74a1b: ending tablet copy session 4015e9df09424332bc3d7a9bad781641-ba183686332c4ab5b606bbc5f1b4e787 on tablet ba183686332c4ab5b606bbc5f1b4e787 with peer 4015e9df09424332bc3d7a9bad781641
I20240501 14:09:49.213718 13728 tablet_copy_service.cc:140] P a278448771e14471ab4df1e6649485bd: Received BeginTabletCopySession request for tablet d948c2d5182b415b8aedb4c7b786247f from peer e26df472efe547f8bc5951dce352471b ({username='slave'} at 127.7.39.134:58033)
I20240501 14:09:49.213862 13728 tablet_copy_service.cc:161] P a278448771e14471ab4df1e6649485bd: Beginning new tablet copy session on tablet d948c2d5182b415b8aedb4c7b786247f from peer e26df472efe547f8bc5951dce352471b at {username='slave'} at 127.7.39.134:58033: session id = e26df472efe547f8bc5951dce352471b-d948c2d5182b415b8aedb4c7b786247f
I20240501 14:09:49.214874 13728 tablet_copy_source_session.cc:205] T d948c2d5182b415b8aedb4c7b786247f P a278448771e14471ab4df1e6649485bd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:49.216114 15479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d948c2d5182b415b8aedb4c7b786247f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:49.218519 15479 tablet_copy_client.cc:783] T d948c2d5182b415b8aedb4c7b786247f P e26df472efe547f8bc5951dce352471b: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:49.218688 15479 tablet_copy_client.cc:647] T d948c2d5182b415b8aedb4c7b786247f P e26df472efe547f8bc5951dce352471b: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:49.221948 14948 raft_consensus.cc:1203] T 0dc94f932e9a4319988e700aff042f22 P e26df472efe547f8bc5951dce352471b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.57->[1.58-1.92] Dedup: 1.80->[1.81-1.92]
I20240501 14:09:49.222337 14816 raft_consensus.cc:1203] T bf0b5fd770b8405b9049c52d5d95bc53 P 4015e9df09424332bc3d7a9bad781641 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.34->[1.35-1.68] Dedup: 1.52->[1.53-1.68]
I20240501 14:09:49.228590 15479 tablet_copy_client.cc:523] T d948c2d5182b415b8aedb4c7b786247f P e26df472efe547f8bc5951dce352471b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:49.230293 15479 tablet_bootstrap.cc:492] T d948c2d5182b415b8aedb4c7b786247f P e26df472efe547f8bc5951dce352471b: Bootstrap starting.
I20240501 14:09:49.232743 13577 consensus_queue.cc:237] T cce2260455ac4edbb9bc54b103c50eb3 P 43736346b1ff48c78ac919c80ba58654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 72, Committed index: 72, Last appended: 1.73, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } }
W20240501 14:09:49.235697 13511 consensus_peers.cc:485] T cce2260455ac4edbb9bc54b103c50eb3 P 43736346b1ff48c78ac919c80ba58654 -> Peer e26df472efe547f8bc5951dce352471b (127.7.39.134:33035): Couldn't send request to peer e26df472efe547f8bc5951dce352471b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cce2260455ac4edbb9bc54b103c50eb3. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:49.236231 13446 raft_consensus.cc:1261] T cce2260455ac4edbb9bc54b103c50eb3 P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Refusing update from remote peer 43736346b1ff48c78ac919c80ba58654: Log matching property violated. Preceding OpId in replica: term: 1 index: 73. Preceding OpId from leader: term: 1 index: 74. (index mismatch)
I20240501 14:09:49.237138 15569 consensus_queue.cc:1035] T cce2260455ac4edbb9bc54b103c50eb3 P 43736346b1ff48c78ac919c80ba58654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 74, Last known committed idx: 73, Time since last communication: 0.000s
I20240501 14:09:49.240155 15212 raft_consensus.cc:2932] T cce2260455ac4edbb9bc54b103c50eb3 P 43736346b1ff48c78ac919c80ba58654 [term 1 LEADER]: Committing config change with OpId 1.74: config changed from index -1 to 74, VOTER e26df472efe547f8bc5951dce352471b (127.7.39.134) added. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } }
I20240501 14:09:49.243248 13304 catalog_manager.cc:5484] T cce2260455ac4edbb9bc54b103c50eb3 P 43736346b1ff48c78ac919c80ba58654 reported cstate change: config changed from index -1 to 74, VOTER e26df472efe547f8bc5951dce352471b (127.7.39.134) added. New cstate: current_term: 1 leader_uuid: "43736346b1ff48c78ac919c80ba58654" committed_config { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } }
I20240501 14:09:49.244952 13443 raft_consensus.cc:2932] T cce2260455ac4edbb9bc54b103c50eb3 P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Committing config change with OpId 1.74: config changed from index -1 to 74, VOTER e26df472efe547f8bc5951dce352471b (127.7.39.134) added. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } }
I20240501 14:09:49.335513 14685 raft_consensus.cc:1203] T 3f34112bba8a43a58420148533a6e989 P 2496e4af45ba458eacaacc9d25dcf01a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.80->[1.81-1.112] Dedup: 1.99->[1.100-1.112]
I20240501 14:09:49.405190 13443 consensus_queue.cc:237] T 341c6b416c374a9ba4fc34d861ba2ef9 P 50d514f034204fc38331aa0a4fd74a1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 64, Committed index: 64, Last appended: 1.64, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } }
W20240501 14:09:49.406611 13380 consensus_peers.cc:485] T 341c6b416c374a9ba4fc34d861ba2ef9 P 50d514f034204fc38331aa0a4fd74a1b -> Peer 4015e9df09424332bc3d7a9bad781641 (127.7.39.133:42905): Couldn't send request to peer 4015e9df09424332bc3d7a9bad781641. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 341c6b416c374a9ba4fc34d861ba2ef9. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:49.410791 13708 raft_consensus.cc:1261] T 341c6b416c374a9ba4fc34d861ba2ef9 P a278448771e14471ab4df1e6649485bd [term 1 FOLLOWER]: Refusing update from remote peer 50d514f034204fc38331aa0a4fd74a1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 64. Preceding OpId from leader: term: 1 index: 65. (index mismatch)
I20240501 14:09:49.415382 15479 tablet_bootstrap.cc:492] T d948c2d5182b415b8aedb4c7b786247f P e26df472efe547f8bc5951dce352471b: Bootstrap replayed 1/1 log segments. Stats: ops{read=102 overwritten=0 applied=102 ignored=0} inserts{seen=733 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:49.415714 15479 tablet_bootstrap.cc:492] T d948c2d5182b415b8aedb4c7b786247f P e26df472efe547f8bc5951dce352471b: Bootstrap complete.
I20240501 14:09:49.415709 15587 consensus_queue.cc:1035] T 341c6b416c374a9ba4fc34d861ba2ef9 P 50d514f034204fc38331aa0a4fd74a1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 65, Last known committed idx: 64, Time since last communication: 0.000s
I20240501 14:09:49.415926 15479 ts_tablet_manager.cc:1367] T d948c2d5182b415b8aedb4c7b786247f P e26df472efe547f8bc5951dce352471b: Time spent bootstrapping tablet: real 0.186s user 0.093s sys 0.005s
I20240501 14:09:49.416409 15479 raft_consensus.cc:348] T d948c2d5182b415b8aedb4c7b786247f P e26df472efe547f8bc5951dce352471b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } }
I20240501 14:09:49.419806 15479 raft_consensus.cc:729] T d948c2d5182b415b8aedb4c7b786247f P e26df472efe547f8bc5951dce352471b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e26df472efe547f8bc5951dce352471b, State: Initialized, Role: FOLLOWER
I20240501 14:09:49.420046 15479 consensus_queue.cc:260] T d948c2d5182b415b8aedb4c7b786247f P e26df472efe547f8bc5951dce352471b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 102, Last appended: 1.102, Last appended by leader: 102, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } }
I20240501 14:09:49.421334 15479 ts_tablet_manager.cc:1398] T d948c2d5182b415b8aedb4c7b786247f P e26df472efe547f8bc5951dce352471b: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:49.421756 13728 tablet_copy_service.cc:342] P a278448771e14471ab4df1e6649485bd: Request end of tablet copy session e26df472efe547f8bc5951dce352471b-d948c2d5182b415b8aedb4c7b786247f received from {username='slave'} at 127.7.39.134:58033
I20240501 14:09:49.421828 13728 tablet_copy_service.cc:433] P a278448771e14471ab4df1e6649485bd: ending tablet copy session e26df472efe547f8bc5951dce352471b-d948c2d5182b415b8aedb4c7b786247f on tablet d948c2d5182b415b8aedb4c7b786247f with peer e26df472efe547f8bc5951dce352471b
I20240501 14:09:49.426682 15587 raft_consensus.cc:2932] T 341c6b416c374a9ba4fc34d861ba2ef9 P 50d514f034204fc38331aa0a4fd74a1b [term 1 LEADER]: Committing config change with OpId 1.65: config changed from index -1 to 65, VOTER 4015e9df09424332bc3d7a9bad781641 (127.7.39.133) added. New config: { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } }
I20240501 14:09:49.428903 13706 raft_consensus.cc:2932] T 341c6b416c374a9ba4fc34d861ba2ef9 P a278448771e14471ab4df1e6649485bd [term 1 FOLLOWER]: Committing config change with OpId 1.65: config changed from index -1 to 65, VOTER 4015e9df09424332bc3d7a9bad781641 (127.7.39.133) added. New config: { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } }
I20240501 14:09:49.431222 13304 catalog_manager.cc:5484] T 341c6b416c374a9ba4fc34d861ba2ef9 P 50d514f034204fc38331aa0a4fd74a1b reported cstate change: config changed from index -1 to 65, VOTER 4015e9df09424332bc3d7a9bad781641 (127.7.39.133) added. New cstate: current_term: 1 leader_uuid: "50d514f034204fc38331aa0a4fd74a1b" committed_config { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } }
I20240501 14:09:49.432256 15434 ts_tablet_manager.cc:902] T 8a89cc8e4b2b476ba954d3cc64537031 P 39bde7c058154f8aa40085db77f8c6e5: Initiating tablet copy from peer 43736346b1ff48c78ac919c80ba58654 (127.7.39.130:43311)
I20240501 14:09:49.432662 15434 tablet_copy_client.cc:308] T 8a89cc8e4b2b476ba954d3cc64537031 P 39bde7c058154f8aa40085db77f8c6e5: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.130:43311
I20240501 14:09:49.433512 13597 tablet_copy_service.cc:140] P 43736346b1ff48c78ac919c80ba58654: Received BeginTabletCopySession request for tablet 8a89cc8e4b2b476ba954d3cc64537031 from peer 39bde7c058154f8aa40085db77f8c6e5 ({username='slave'} at 127.7.39.135:52733)
I20240501 14:09:49.433620 13597 tablet_copy_service.cc:161] P 43736346b1ff48c78ac919c80ba58654: Beginning new tablet copy session on tablet 8a89cc8e4b2b476ba954d3cc64537031 from peer 39bde7c058154f8aa40085db77f8c6e5 at {username='slave'} at 127.7.39.135:52733: session id = 39bde7c058154f8aa40085db77f8c6e5-8a89cc8e4b2b476ba954d3cc64537031
I20240501 14:09:49.434443 13597 tablet_copy_source_session.cc:205] T 8a89cc8e4b2b476ba954d3cc64537031 P 43736346b1ff48c78ac919c80ba58654: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:49.435431 15434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a89cc8e4b2b476ba954d3cc64537031. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:49.437438 15434 tablet_copy_client.cc:783] T 8a89cc8e4b2b476ba954d3cc64537031 P 39bde7c058154f8aa40085db77f8c6e5: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:49.437573 15434 tablet_copy_client.cc:647] T 8a89cc8e4b2b476ba954d3cc64537031 P 39bde7c058154f8aa40085db77f8c6e5: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:49.438458 14815 raft_consensus.cc:1203] T ba183686332c4ab5b606bbc5f1b4e787 P 4015e9df09424332bc3d7a9bad781641 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.57->[1.58-1.78] Dedup: 1.70->[1.71-1.78]
I20240501 14:09:49.439173 15505 ts_tablet_manager.cc:902] T 4d3b3213e1dd426b8efc3fdc733b1842 P 2496e4af45ba458eacaacc9d25dcf01a: Initiating tablet copy from peer a278448771e14471ab4df1e6649485bd (127.7.39.131:46461)
I20240501 14:09:49.439662 15505 tablet_copy_client.cc:308] T 4d3b3213e1dd426b8efc3fdc733b1842 P 2496e4af45ba458eacaacc9d25dcf01a: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.131:46461
I20240501 14:09:49.440562 13728 tablet_copy_service.cc:140] P a278448771e14471ab4df1e6649485bd: Received BeginTabletCopySession request for tablet 4d3b3213e1dd426b8efc3fdc733b1842 from peer 2496e4af45ba458eacaacc9d25dcf01a ({username='slave'} at 127.7.39.132:54871)
I20240501 14:09:49.440654 13728 tablet_copy_service.cc:161] P a278448771e14471ab4df1e6649485bd: Beginning new tablet copy session on tablet 4d3b3213e1dd426b8efc3fdc733b1842 from peer 2496e4af45ba458eacaacc9d25dcf01a at {username='slave'} at 127.7.39.132:54871: session id = 2496e4af45ba458eacaacc9d25dcf01a-4d3b3213e1dd426b8efc3fdc733b1842
I20240501 14:09:49.441462 13728 tablet_copy_source_session.cc:205] T 4d3b3213e1dd426b8efc3fdc733b1842 P a278448771e14471ab4df1e6649485bd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:49.444644 15505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d3b3213e1dd426b8efc3fdc733b1842. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:49.446950 15505 tablet_copy_client.cc:783] T 4d3b3213e1dd426b8efc3fdc733b1842 P 2496e4af45ba458eacaacc9d25dcf01a: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:49.453961 15434 tablet_copy_client.cc:523] T 8a89cc8e4b2b476ba954d3cc64537031 P 39bde7c058154f8aa40085db77f8c6e5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:49.454387 15505 tablet_copy_client.cc:647] T 4d3b3213e1dd426b8efc3fdc733b1842 P 2496e4af45ba458eacaacc9d25dcf01a: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:49.455399 15434 tablet_bootstrap.cc:492] T 8a89cc8e4b2b476ba954d3cc64537031 P 39bde7c058154f8aa40085db77f8c6e5: Bootstrap starting.
I20240501 14:09:49.460306 15505 tablet_copy_client.cc:523] T 4d3b3213e1dd426b8efc3fdc733b1842 P 2496e4af45ba458eacaacc9d25dcf01a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:49.461699 15505 tablet_bootstrap.cc:492] T 4d3b3213e1dd426b8efc3fdc733b1842 P 2496e4af45ba458eacaacc9d25dcf01a: Bootstrap starting.
I20240501 14:09:49.549660 15082 raft_consensus.cc:1203] T e4266d76760c4f84b16991f8aa38a3c3 P 39bde7c058154f8aa40085db77f8c6e5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.44->[1.45-1.71] Dedup: 1.61->[1.62-1.71]
I20240501 14:09:49.560982 13707 consensus_queue.cc:237] T bd836361783c4b3782c0f64c6794488b P a278448771e14471ab4df1e6649485bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 1.74, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "2496e4af45ba458eacaacc9d25dcf01a" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 43977 } }
W20240501 14:09:49.565237 13643 consensus_peers.cc:485] T bd836361783c4b3782c0f64c6794488b P a278448771e14471ab4df1e6649485bd -> Peer 2496e4af45ba458eacaacc9d25dcf01a (127.7.39.132:43977): Couldn't send request to peer 2496e4af45ba458eacaacc9d25dcf01a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bd836361783c4b3782c0f64c6794488b. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:49.566082 13446 raft_consensus.cc:1261] T bd836361783c4b3782c0f64c6794488b P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Refusing update from remote peer a278448771e14471ab4df1e6649485bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 74. Preceding OpId from leader: term: 1 index: 75. (index mismatch)
I20240501 14:09:49.566541 15538 consensus_queue.cc:1035] T bd836361783c4b3782c0f64c6794488b P a278448771e14471ab4df1e6649485bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 75, Last known committed idx: 74, Time since last communication: 0.000s
I20240501 14:09:49.572412 15000 raft_consensus.cc:2932] T bd836361783c4b3782c0f64c6794488b P a278448771e14471ab4df1e6649485bd [term 1 LEADER]: Committing config change with OpId 1.75: config changed from index -1 to 75, VOTER 2496e4af45ba458eacaacc9d25dcf01a (127.7.39.132) added. New config: { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "2496e4af45ba458eacaacc9d25dcf01a" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 43977 } } }
I20240501 14:09:49.575891 13299 catalog_manager.cc:5484] T bd836361783c4b3782c0f64c6794488b P a278448771e14471ab4df1e6649485bd reported cstate change: config changed from index -1 to 75, VOTER 2496e4af45ba458eacaacc9d25dcf01a (127.7.39.132) added. New cstate: current_term: 1 leader_uuid: "a278448771e14471ab4df1e6649485bd" committed_config { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "2496e4af45ba458eacaacc9d25dcf01a" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 43977 } } }
I20240501 14:09:49.583240 13444 raft_consensus.cc:2932] T bd836361783c4b3782c0f64c6794488b P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Committing config change with OpId 1.75: config changed from index -1 to 75, VOTER 2496e4af45ba458eacaacc9d25dcf01a (127.7.39.132) added. New config: { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "2496e4af45ba458eacaacc9d25dcf01a" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 43977 } } }
I20240501 14:09:49.653707 14948 raft_consensus.cc:1203] T d948c2d5182b415b8aedb4c7b786247f P e26df472efe547f8bc5951dce352471b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.78->[1.79-1.118] Dedup: 1.102->[1.103-1.118]
I20240501 14:09:49.665616 15434 tablet_bootstrap.cc:492] T 8a89cc8e4b2b476ba954d3cc64537031 P 39bde7c058154f8aa40085db77f8c6e5: Bootstrap replayed 1/1 log segments. Stats: ops{read=106 overwritten=0 applied=106 ignored=0} inserts{seen=791 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:49.666069 15434 tablet_bootstrap.cc:492] T 8a89cc8e4b2b476ba954d3cc64537031 P 39bde7c058154f8aa40085db77f8c6e5: Bootstrap complete.
I20240501 14:09:49.666366 15434 ts_tablet_manager.cc:1367] T 8a89cc8e4b2b476ba954d3cc64537031 P 39bde7c058154f8aa40085db77f8c6e5: Time spent bootstrapping tablet: real 0.211s user 0.083s sys 0.021s
I20240501 14:09:49.666915 15434 raft_consensus.cc:348] T 8a89cc8e4b2b476ba954d3cc64537031 P 39bde7c058154f8aa40085db77f8c6e5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } }
I20240501 14:09:49.667150 15434 raft_consensus.cc:729] T 8a89cc8e4b2b476ba954d3cc64537031 P 39bde7c058154f8aa40085db77f8c6e5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39bde7c058154f8aa40085db77f8c6e5, State: Initialized, Role: FOLLOWER
I20240501 14:09:49.667382 15434 consensus_queue.cc:260] T 8a89cc8e4b2b476ba954d3cc64537031 P 39bde7c058154f8aa40085db77f8c6e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 106, Last appended: 1.106, Last appended by leader: 106, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } }
I20240501 14:09:49.668666 15434 ts_tablet_manager.cc:1398] T 8a89cc8e4b2b476ba954d3cc64537031 P 39bde7c058154f8aa40085db77f8c6e5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:49.670928 13597 tablet_copy_service.cc:342] P 43736346b1ff48c78ac919c80ba58654: Request end of tablet copy session 39bde7c058154f8aa40085db77f8c6e5-8a89cc8e4b2b476ba954d3cc64537031 received from {username='slave'} at 127.7.39.135:52733
I20240501 14:09:49.671010 13597 tablet_copy_service.cc:433] P 43736346b1ff48c78ac919c80ba58654: ending tablet copy session 39bde7c058154f8aa40085db77f8c6e5-8a89cc8e4b2b476ba954d3cc64537031 on tablet 8a89cc8e4b2b476ba954d3cc64537031 with peer 39bde7c058154f8aa40085db77f8c6e5
I20240501 14:09:49.745141 15479 ts_tablet_manager.cc:902] T cce2260455ac4edbb9bc54b103c50eb3 P e26df472efe547f8bc5951dce352471b: Initiating tablet copy from peer 43736346b1ff48c78ac919c80ba58654 (127.7.39.130:43311)
I20240501 14:09:49.745715 15479 tablet_copy_client.cc:308] T cce2260455ac4edbb9bc54b103c50eb3 P e26df472efe547f8bc5951dce352471b: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.130:43311
I20240501 14:09:49.748991 13597 tablet_copy_service.cc:140] P 43736346b1ff48c78ac919c80ba58654: Received BeginTabletCopySession request for tablet cce2260455ac4edbb9bc54b103c50eb3 from peer e26df472efe547f8bc5951dce352471b ({username='slave'} at 127.7.39.134:45565)
I20240501 14:09:49.749104 13597 tablet_copy_service.cc:161] P 43736346b1ff48c78ac919c80ba58654: Beginning new tablet copy session on tablet cce2260455ac4edbb9bc54b103c50eb3 from peer e26df472efe547f8bc5951dce352471b at {username='slave'} at 127.7.39.134:45565: session id = e26df472efe547f8bc5951dce352471b-cce2260455ac4edbb9bc54b103c50eb3
I20240501 14:09:49.750052 13597 tablet_copy_source_session.cc:205] T cce2260455ac4edbb9bc54b103c50eb3 P 43736346b1ff48c78ac919c80ba58654: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:49.766808 15505 tablet_bootstrap.cc:492] T 4d3b3213e1dd426b8efc3fdc733b1842 P 2496e4af45ba458eacaacc9d25dcf01a: Bootstrap replayed 1/1 log segments. Stats: ops{read=91 overwritten=0 applied=91 ignored=0} inserts{seen=722 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:49.767572 15479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cce2260455ac4edbb9bc54b103c50eb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:49.769896 15479 tablet_copy_client.cc:783] T cce2260455ac4edbb9bc54b103c50eb3 P e26df472efe547f8bc5951dce352471b: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:49.770056 15479 tablet_copy_client.cc:647] T cce2260455ac4edbb9bc54b103c50eb3 P e26df472efe547f8bc5951dce352471b: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:49.773150 15505 tablet_bootstrap.cc:492] T 4d3b3213e1dd426b8efc3fdc733b1842 P 2496e4af45ba458eacaacc9d25dcf01a: Bootstrap complete.
I20240501 14:09:49.774087 15505 ts_tablet_manager.cc:1367] T 4d3b3213e1dd426b8efc3fdc733b1842 P 2496e4af45ba458eacaacc9d25dcf01a: Time spent bootstrapping tablet: real 0.312s user 0.091s sys 0.003s
I20240501 14:09:49.774642 15505 raft_consensus.cc:348] T 4d3b3213e1dd426b8efc3fdc733b1842 P 2496e4af45ba458eacaacc9d25dcf01a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "2496e4af45ba458eacaacc9d25dcf01a" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 43977 } }
I20240501 14:09:49.774878 15505 raft_consensus.cc:729] T 4d3b3213e1dd426b8efc3fdc733b1842 P 2496e4af45ba458eacaacc9d25dcf01a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2496e4af45ba458eacaacc9d25dcf01a, State: Initialized, Role: FOLLOWER
I20240501 14:09:49.775117 15505 consensus_queue.cc:260] T 4d3b3213e1dd426b8efc3fdc733b1842 P 2496e4af45ba458eacaacc9d25dcf01a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 91, Last appended: 1.91, Last appended by leader: 91, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "2496e4af45ba458eacaacc9d25dcf01a" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 43977 } }
I20240501 14:09:49.775513 15505 ts_tablet_manager.cc:1398] T 4d3b3213e1dd426b8efc3fdc733b1842 P 2496e4af45ba458eacaacc9d25dcf01a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:49.776227 13728 tablet_copy_service.cc:342] P a278448771e14471ab4df1e6649485bd: Request end of tablet copy session 2496e4af45ba458eacaacc9d25dcf01a-4d3b3213e1dd426b8efc3fdc733b1842 received from {username='slave'} at 127.7.39.132:54871
I20240501 14:09:49.776304 13728 tablet_copy_service.cc:433] P a278448771e14471ab4df1e6649485bd: ending tablet copy session 2496e4af45ba458eacaacc9d25dcf01a-4d3b3213e1dd426b8efc3fdc733b1842 on tablet 4d3b3213e1dd426b8efc3fdc733b1842 with peer 2496e4af45ba458eacaacc9d25dcf01a
I20240501 14:09:49.786955 15479 tablet_copy_client.cc:523] T cce2260455ac4edbb9bc54b103c50eb3 P e26df472efe547f8bc5951dce352471b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:49.788486 15479 tablet_bootstrap.cc:492] T cce2260455ac4edbb9bc54b103c50eb3 P e26df472efe547f8bc5951dce352471b: Bootstrap starting.
I20240501 14:09:49.854574 15081 raft_consensus.cc:1203] T 8a89cc8e4b2b476ba954d3cc64537031 P 39bde7c058154f8aa40085db77f8c6e5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.95->[1.96-1.120] Dedup: 1.106->[1.107-1.120]
I20240501 14:09:49.890082 14665 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:49.911664 13555 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:49.927964 15479 tablet_bootstrap.cc:492] T cce2260455ac4edbb9bc54b103c50eb3 P e26df472efe547f8bc5951dce352471b: Bootstrap replayed 1/1 log segments. Stats: ops{read=84 overwritten=0 applied=84 ignored=0} inserts{seen=756 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:49.928833 15479 tablet_bootstrap.cc:492] T cce2260455ac4edbb9bc54b103c50eb3 P e26df472efe547f8bc5951dce352471b: Bootstrap complete.
I20240501 14:09:49.930166 15479 ts_tablet_manager.cc:1367] T cce2260455ac4edbb9bc54b103c50eb3 P e26df472efe547f8bc5951dce352471b: Time spent bootstrapping tablet: real 0.142s user 0.089s sys 0.001s
I20240501 14:09:49.930929 15479 raft_consensus.cc:348] T cce2260455ac4edbb9bc54b103c50eb3 P e26df472efe547f8bc5951dce352471b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } }
I20240501 14:09:49.931185 15479 raft_consensus.cc:729] T cce2260455ac4edbb9bc54b103c50eb3 P e26df472efe547f8bc5951dce352471b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e26df472efe547f8bc5951dce352471b, State: Initialized, Role: FOLLOWER
I20240501 14:09:49.931421 15479 consensus_queue.cc:260] T cce2260455ac4edbb9bc54b103c50eb3 P e26df472efe547f8bc5951dce352471b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 84, Last appended: 1.84, Last appended by leader: 84, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } }
I20240501 14:09:49.932157 15479 ts_tablet_manager.cc:1398] T cce2260455ac4edbb9bc54b103c50eb3 P e26df472efe547f8bc5951dce352471b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:49.937281 13597 tablet_copy_service.cc:342] P 43736346b1ff48c78ac919c80ba58654: Request end of tablet copy session e26df472efe547f8bc5951dce352471b-cce2260455ac4edbb9bc54b103c50eb3 received from {username='slave'} at 127.7.39.134:45565
I20240501 14:09:49.937381 13597 tablet_copy_service.cc:433] P 43736346b1ff48c78ac919c80ba58654: ending tablet copy session e26df472efe547f8bc5951dce352471b-cce2260455ac4edbb9bc54b103c50eb3 on tablet cce2260455ac4edbb9bc54b103c50eb3 with peer e26df472efe547f8bc5951dce352471b
I20240501 14:09:49.948437 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:49.950667 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:49.962263 14684 raft_consensus.cc:1203] T 4d3b3213e1dd426b8efc3fdc733b1842 P 2496e4af45ba458eacaacc9d25dcf01a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.79->[1.80-1.105] Dedup: 1.91->[1.92-1.105]
I20240501 14:09:49.962478 13687 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:49.970118 15505 ts_tablet_manager.cc:902] T bd836361783c4b3782c0f64c6794488b P 2496e4af45ba458eacaacc9d25dcf01a: Initiating tablet copy from peer a278448771e14471ab4df1e6649485bd (127.7.39.131:46461)
I20240501 14:09:49.979215 15505 tablet_copy_client.cc:308] T bd836361783c4b3782c0f64c6794488b P 2496e4af45ba458eacaacc9d25dcf01a: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.131:46461
I20240501 14:09:49.979718 13728 tablet_copy_service.cc:140] P a278448771e14471ab4df1e6649485bd: Received BeginTabletCopySession request for tablet bd836361783c4b3782c0f64c6794488b from peer 2496e4af45ba458eacaacc9d25dcf01a ({username='slave'} at 127.7.39.132:54871)
I20240501 14:09:49.979830 13728 tablet_copy_service.cc:161] P a278448771e14471ab4df1e6649485bd: Beginning new tablet copy session on tablet bd836361783c4b3782c0f64c6794488b from peer 2496e4af45ba458eacaacc9d25dcf01a at {username='slave'} at 127.7.39.132:54871: session id = 2496e4af45ba458eacaacc9d25dcf01a-bd836361783c4b3782c0f64c6794488b
I20240501 14:09:49.980770 13728 tablet_copy_source_session.cc:205] T bd836361783c4b3782c0f64c6794488b P a278448771e14471ab4df1e6649485bd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:49.985158 15505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd836361783c4b3782c0f64c6794488b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:49.986419 15696 ts_tablet_manager.cc:902] T 341c6b416c374a9ba4fc34d861ba2ef9 P 4015e9df09424332bc3d7a9bad781641: Initiating tablet copy from peer 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129:37933)
I20240501 14:09:49.995815 15062 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:50.004215 14928 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:50.015352 15505 tablet_copy_client.cc:783] T bd836361783c4b3782c0f64c6794488b P 2496e4af45ba458eacaacc9d25dcf01a: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:50.015623 15505 tablet_copy_client.cc:647] T bd836361783c4b3782c0f64c6794488b P 2496e4af45ba458eacaacc9d25dcf01a: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:50.016423 15696 tablet_copy_client.cc:308] T 341c6b416c374a9ba4fc34d861ba2ef9 P 4015e9df09424332bc3d7a9bad781641: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.129:37933
I20240501 14:09:50.018059 13466 tablet_copy_service.cc:140] P 50d514f034204fc38331aa0a4fd74a1b: Received BeginTabletCopySession request for tablet 341c6b416c374a9ba4fc34d861ba2ef9 from peer 4015e9df09424332bc3d7a9bad781641 ({username='slave'} at 127.7.39.133:33105)
I20240501 14:09:50.018151 13466 tablet_copy_service.cc:161] P 50d514f034204fc38331aa0a4fd74a1b: Beginning new tablet copy session on tablet 341c6b416c374a9ba4fc34d861ba2ef9 from peer 4015e9df09424332bc3d7a9bad781641 at {username='slave'} at 127.7.39.133:33105: session id = 4015e9df09424332bc3d7a9bad781641-341c6b416c374a9ba4fc34d861ba2ef9
I20240501 14:09:50.019006 13466 tablet_copy_source_session.cc:205] T 341c6b416c374a9ba4fc34d861ba2ef9 P 50d514f034204fc38331aa0a4fd74a1b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:50.024950 15505 tablet_copy_client.cc:523] T bd836361783c4b3782c0f64c6794488b P 2496e4af45ba458eacaacc9d25dcf01a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:50.026580 15505 tablet_bootstrap.cc:492] T bd836361783c4b3782c0f64c6794488b P 2496e4af45ba458eacaacc9d25dcf01a: Bootstrap starting.
I20240501 14:09:50.038707 15696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 341c6b416c374a9ba4fc34d861ba2ef9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:50.061983 15696 tablet_copy_client.cc:783] T 341c6b416c374a9ba4fc34d861ba2ef9 P 4015e9df09424332bc3d7a9bad781641: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:50.068442 15696 tablet_copy_client.cc:647] T 341c6b416c374a9ba4fc34d861ba2ef9 P 4015e9df09424332bc3d7a9bad781641: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:50.080482 15696 tablet_copy_client.cc:523] T 341c6b416c374a9ba4fc34d861ba2ef9 P 4015e9df09424332bc3d7a9bad781641: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:50.082182 15696 tablet_bootstrap.cc:492] T 341c6b416c374a9ba4fc34d861ba2ef9 P 4015e9df09424332bc3d7a9bad781641: Bootstrap starting.
I20240501 14:09:50.247202 14947 raft_consensus.cc:1203] T cce2260455ac4edbb9bc54b103c50eb3 P e26df472efe547f8bc5951dce352471b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.73->[1.74-1.94] Dedup: 1.84->[1.85-1.94]
I20240501 14:09:50.269563 15696 tablet_bootstrap.cc:492] T 341c6b416c374a9ba4fc34d861ba2ef9 P 4015e9df09424332bc3d7a9bad781641: Bootstrap replayed 1/1 log segments. Stats: ops{read=81 overwritten=0 applied=80 ignored=0} inserts{seen=765 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 14:09:50.270011 15696 tablet_bootstrap.cc:492] T 341c6b416c374a9ba4fc34d861ba2ef9 P 4015e9df09424332bc3d7a9bad781641: Bootstrap complete.
I20240501 14:09:50.270242 15696 ts_tablet_manager.cc:1367] T 341c6b416c374a9ba4fc34d861ba2ef9 P 4015e9df09424332bc3d7a9bad781641: Time spent bootstrapping tablet: real 0.188s user 0.070s sys 0.010s
I20240501 14:09:50.270691 15696 raft_consensus.cc:348] T 341c6b416c374a9ba4fc34d861ba2ef9 P 4015e9df09424332bc3d7a9bad781641 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } }
I20240501 14:09:50.271296 15696 raft_consensus.cc:729] T 341c6b416c374a9ba4fc34d861ba2ef9 P 4015e9df09424332bc3d7a9bad781641 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4015e9df09424332bc3d7a9bad781641, State: Initialized, Role: FOLLOWER
I20240501 14:09:50.271468 15696 consensus_queue.cc:260] T 341c6b416c374a9ba4fc34d861ba2ef9 P 4015e9df09424332bc3d7a9bad781641 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 80, Last appended: 1.81, Last appended by leader: 81, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } }
I20240501 14:09:50.271917 15696 ts_tablet_manager.cc:1398] T 341c6b416c374a9ba4fc34d861ba2ef9 P 4015e9df09424332bc3d7a9bad781641: Time spent starting tablet: real 0.002s user 0.001s sys 0.004s
I20240501 14:09:50.273414 13466 tablet_copy_service.cc:342] P 50d514f034204fc38331aa0a4fd74a1b: Request end of tablet copy session 4015e9df09424332bc3d7a9bad781641-341c6b416c374a9ba4fc34d861ba2ef9 received from {username='slave'} at 127.7.39.133:33105
I20240501 14:09:50.273490 13466 tablet_copy_service.cc:433] P 50d514f034204fc38331aa0a4fd74a1b: ending tablet copy session 4015e9df09424332bc3d7a9bad781641-341c6b416c374a9ba4fc34d861ba2ef9 on tablet 341c6b416c374a9ba4fc34d861ba2ef9 with peer 4015e9df09424332bc3d7a9bad781641
I20240501 14:09:50.290047 15505 tablet_bootstrap.cc:492] T bd836361783c4b3782c0f64c6794488b P 2496e4af45ba458eacaacc9d25dcf01a: Bootstrap replayed 1/1 log segments. Stats: ops{read=84 overwritten=0 applied=84 ignored=0} inserts{seen=715 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:50.290400 15505 tablet_bootstrap.cc:492] T bd836361783c4b3782c0f64c6794488b P 2496e4af45ba458eacaacc9d25dcf01a: Bootstrap complete.
I20240501 14:09:50.290620 15505 ts_tablet_manager.cc:1367] T bd836361783c4b3782c0f64c6794488b P 2496e4af45ba458eacaacc9d25dcf01a: Time spent bootstrapping tablet: real 0.264s user 0.074s sys 0.010s
I20240501 14:09:50.291141 15505 raft_consensus.cc:348] T bd836361783c4b3782c0f64c6794488b P 2496e4af45ba458eacaacc9d25dcf01a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "2496e4af45ba458eacaacc9d25dcf01a" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 43977 } }
I20240501 14:09:50.291262 15505 raft_consensus.cc:729] T bd836361783c4b3782c0f64c6794488b P 2496e4af45ba458eacaacc9d25dcf01a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2496e4af45ba458eacaacc9d25dcf01a, State: Initialized, Role: FOLLOWER
I20240501 14:09:50.291425 15505 consensus_queue.cc:260] T bd836361783c4b3782c0f64c6794488b P 2496e4af45ba458eacaacc9d25dcf01a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 84, Last appended: 1.84, Last appended by leader: 84, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "2496e4af45ba458eacaacc9d25dcf01a" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 43977 } }
I20240501 14:09:50.297132 15505 ts_tablet_manager.cc:1398] T bd836361783c4b3782c0f64c6794488b P 2496e4af45ba458eacaacc9d25dcf01a: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:50.300899 13728 tablet_copy_service.cc:342] P a278448771e14471ab4df1e6649485bd: Request end of tablet copy session 2496e4af45ba458eacaacc9d25dcf01a-bd836361783c4b3782c0f64c6794488b received from {username='slave'} at 127.7.39.132:54871
I20240501 14:09:50.300985 13728 tablet_copy_service.cc:433] P a278448771e14471ab4df1e6649485bd: ending tablet copy session 2496e4af45ba458eacaacc9d25dcf01a-bd836361783c4b3782c0f64c6794488b on tablet bd836361783c4b3782c0f64c6794488b with peer 2496e4af45ba458eacaacc9d25dcf01a
I20240501 14:09:50.427356 14684 raft_consensus.cc:1203] T bd836361783c4b3782c0f64c6794488b P 2496e4af45ba458eacaacc9d25dcf01a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.74->[1.75-1.103] Dedup: 1.84->[1.85-1.103]
I20240501 14:09:50.572439 14815 raft_consensus.cc:1203] T 341c6b416c374a9ba4fc34d861ba2ef9 P 4015e9df09424332bc3d7a9bad781641 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.64->[1.65-1.93] Dedup: 1.81->[1.82-1.93]
I20240501 14:09:50.622732 14665 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:50.626510 15062 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:50.652597 13555 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:50.659016 14928 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:50.709416 13687 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:50.786761 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:50.801131 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:51.296672 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:51.312189 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:51.333899 14665 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:51.346956 15062 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:51.349470 13555 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:51.431123 14928 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:51.445905 13687 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:51.814682 13707 consensus_queue.cc:237] T bd836361783c4b3782c0f64c6794488b P a278448771e14471ab4df1e6649485bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 126, Committed index: 126, Last appended: 1.126, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 127 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "2496e4af45ba458eacaacc9d25dcf01a" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 43977 } }
I20240501 14:09:51.826239 14684 raft_consensus.cc:1261] T bd836361783c4b3782c0f64c6794488b P 2496e4af45ba458eacaacc9d25dcf01a [term 1 FOLLOWER]: Refusing update from remote peer a278448771e14471ab4df1e6649485bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 126. Preceding OpId from leader: term: 1 index: 127. (index mismatch)
I20240501 14:09:51.826750 14601 consensus_queue.cc:1035] T bd836361783c4b3782c0f64c6794488b P a278448771e14471ab4df1e6649485bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "2496e4af45ba458eacaacc9d25dcf01a" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 127, Last known committed idx: 126, Time since last communication: 0.000s
I20240501 14:09:51.830067 14601 raft_consensus.cc:2932] T bd836361783c4b3782c0f64c6794488b P a278448771e14471ab4df1e6649485bd [term 1 LEADER]: Committing config change with OpId 1.127: config changed from index 75 to 127, VOTER 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129) evicted. New config: { opid_index: 127 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "2496e4af45ba458eacaacc9d25dcf01a" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 43977 } } }
I20240501 14:09:51.833101 14684 raft_consensus.cc:2932] T bd836361783c4b3782c0f64c6794488b P 2496e4af45ba458eacaacc9d25dcf01a [term 1 FOLLOWER]: Committing config change with OpId 1.127: config changed from index 75 to 127, VOTER 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129) evicted. New config: { opid_index: 127 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "2496e4af45ba458eacaacc9d25dcf01a" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 43977 } } }
I20240501 14:09:51.833360 13299 catalog_manager.cc:5484] T bd836361783c4b3782c0f64c6794488b P a278448771e14471ab4df1e6649485bd reported cstate change: config changed from index 75 to 127, VOTER 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129) evicted. New cstate: current_term: 1 leader_uuid: "a278448771e14471ab4df1e6649485bd" committed_config { opid_index: 127 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "2496e4af45ba458eacaacc9d25dcf01a" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 43977 } } }
I20240501 14:09:51.873925 13426 tablet_service.cc:1514] Processing DeleteTablet for tablet bd836361783c4b3782c0f64c6794488b with delete_type TABLET_DATA_TOMBSTONED (TS 50d514f034204fc38331aa0a4fd74a1b not found in new config with opid_index 127) from {username='slave'} at 127.0.0.1:45262
I20240501 14:09:51.906848 15853 tablet_replica.cc:330] T bd836361783c4b3782c0f64c6794488b P 50d514f034204fc38331aa0a4fd74a1b: stopping tablet replica
I20240501 14:09:51.907241 15853 raft_consensus.cc:2229] T bd836361783c4b3782c0f64c6794488b P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:51.907431 15853 raft_consensus.cc:2258] T bd836361783c4b3782c0f64c6794488b P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:51.914631 15853 ts_tablet_manager.cc:1875] T bd836361783c4b3782c0f64c6794488b P 50d514f034204fc38331aa0a4fd74a1b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:51.926894 15853 ts_tablet_manager.cc:1888] T bd836361783c4b3782c0f64c6794488b P 50d514f034204fc38331aa0a4fd74a1b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.126
I20240501 14:09:51.927029 15853 log.cc:1196] T bd836361783c4b3782c0f64c6794488b P 50d514f034204fc38331aa0a4fd74a1b: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal/wals/bd836361783c4b3782c0f64c6794488b
I20240501 14:09:51.928190 13291 catalog_manager.cc:4830] TS 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129:37933): tablet bd836361783c4b3782c0f64c6794488b (table rebalance_test_table_0 [id=b91b950953be4c81b3156e9c2c4e50ee]) successfully deleted
I20240501 14:09:52.161523 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:52.171237 15062 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:52.182042 14665 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:52.190649 13687 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:52.257453 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:52.277416 14928 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:52.292857 13555 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:52.705188 13577 consensus_queue.cc:237] T 8a89cc8e4b2b476ba954d3cc64537031 P 43736346b1ff48c78ac919c80ba58654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 197, Committed index: 197, Last appended: 1.197, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 198 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } }
I20240501 14:09:52.710898 15081 raft_consensus.cc:1261] T 8a89cc8e4b2b476ba954d3cc64537031 P 39bde7c058154f8aa40085db77f8c6e5 [term 1 FOLLOWER]: Refusing update from remote peer 43736346b1ff48c78ac919c80ba58654: Log matching property violated. Preceding OpId in replica: term: 1 index: 197. Preceding OpId from leader: term: 1 index: 199. (index mismatch)
I20240501 14:09:52.715456 15233 consensus_queue.cc:1035] T 8a89cc8e4b2b476ba954d3cc64537031 P 43736346b1ff48c78ac919c80ba58654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 198, Last known committed idx: 197, Time since last communication: 0.000s
I20240501 14:09:52.727849 15212 raft_consensus.cc:2932] T 8a89cc8e4b2b476ba954d3cc64537031 P 43736346b1ff48c78ac919c80ba58654 [term 1 LEADER]: Committing config change with OpId 1.198: config changed from index 96 to 198, VOTER a278448771e14471ab4df1e6649485bd (127.7.39.131) evicted. New config: { opid_index: 198 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } } }
I20240501 14:09:52.728991 15081 raft_consensus.cc:2932] T 8a89cc8e4b2b476ba954d3cc64537031 P 39bde7c058154f8aa40085db77f8c6e5 [term 1 FOLLOWER]: Committing config change with OpId 1.198: config changed from index 96 to 198, VOTER a278448771e14471ab4df1e6649485bd (127.7.39.131) evicted. New config: { opid_index: 198 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } } }
I20240501 14:09:52.734087 13301 catalog_manager.cc:5484] T 8a89cc8e4b2b476ba954d3cc64537031 P 43736346b1ff48c78ac919c80ba58654 reported cstate change: config changed from index 96 to 198, VOTER a278448771e14471ab4df1e6649485bd (127.7.39.131) evicted. New cstate: current_term: 1 leader_uuid: "43736346b1ff48c78ac919c80ba58654" committed_config { opid_index: 198 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } } }
I20240501 14:09:52.831130 13687 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a89cc8e4b2b476ba954d3cc64537031 with delete_type TABLET_DATA_TOMBSTONED (TS a278448771e14471ab4df1e6649485bd not found in new config with opid_index 198) from {username='slave'} at 127.0.0.1:45074
I20240501 14:09:52.831702 15930 tablet_replica.cc:330] T 8a89cc8e4b2b476ba954d3cc64537031 P a278448771e14471ab4df1e6649485bd: stopping tablet replica
I20240501 14:09:52.832089 15930 raft_consensus.cc:2229] T 8a89cc8e4b2b476ba954d3cc64537031 P a278448771e14471ab4df1e6649485bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:52.832341 15930 raft_consensus.cc:2258] T 8a89cc8e4b2b476ba954d3cc64537031 P a278448771e14471ab4df1e6649485bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:52.833767 15930 ts_tablet_manager.cc:1875] T 8a89cc8e4b2b476ba954d3cc64537031 P a278448771e14471ab4df1e6649485bd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:52.849484 15930 ts_tablet_manager.cc:1888] T 8a89cc8e4b2b476ba954d3cc64537031 P a278448771e14471ab4df1e6649485bd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.197
I20240501 14:09:52.849694 15930 log.cc:1196] T 8a89cc8e4b2b476ba954d3cc64537031 P a278448771e14471ab4df1e6649485bd: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal/wals/8a89cc8e4b2b476ba954d3cc64537031
I20240501 14:09:52.854074 13288 catalog_manager.cc:4830] TS a278448771e14471ab4df1e6649485bd (127.7.39.131:46461): tablet 8a89cc8e4b2b476ba954d3cc64537031 (table rebalance_test_table_1 [id=b18f113d94ba4379ab99c6c63917bc1a]) successfully deleted
I20240501 14:09:53.257524 14665 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:53.275779 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:53.284317 13687 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:53.304692 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:53.308507 15062 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:53.309993 13555 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:53.357650 14928 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:53.640169 13446 consensus_queue.cc:237] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 1.205, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 206 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } }
I20240501 14:09:53.645841 14814 raft_consensus.cc:1261] T ba183686332c4ab5b606bbc5f1b4e787 P 4015e9df09424332bc3d7a9bad781641 [term 1 FOLLOWER]: Refusing update from remote peer 50d514f034204fc38331aa0a4fd74a1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 1 index: 206. (index mismatch)
I20240501 14:09:53.646344 15587 consensus_queue.cc:1035] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
I20240501 14:09:53.649335 15393 raft_consensus.cc:2932] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b [term 1 LEADER]: Committing config change with OpId 1.206: config changed from index 58 to 206, VOTER a278448771e14471ab4df1e6649485bd (127.7.39.131) evicted. New config: { opid_index: 206 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } }
I20240501 14:09:53.651011 14816 raft_consensus.cc:2932] T ba183686332c4ab5b606bbc5f1b4e787 P 4015e9df09424332bc3d7a9bad781641 [term 1 FOLLOWER]: Committing config change with OpId 1.206: config changed from index 58 to 206, VOTER a278448771e14471ab4df1e6649485bd (127.7.39.131) evicted. New config: { opid_index: 206 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } }
I20240501 14:09:53.653712 13302 catalog_manager.cc:5484] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b reported cstate change: config changed from index 58 to 206, VOTER a278448771e14471ab4df1e6649485bd (127.7.39.131) evicted. New cstate: current_term: 1 leader_uuid: "50d514f034204fc38331aa0a4fd74a1b" committed_config { opid_index: 206 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } }
I20240501 14:09:53.711441 13687 tablet_service.cc:1514] Processing DeleteTablet for tablet ba183686332c4ab5b606bbc5f1b4e787 with delete_type TABLET_DATA_TOMBSTONED (TS a278448771e14471ab4df1e6649485bd not found in new config with opid_index 206) from {username='slave'} at 127.0.0.1:45074
I20240501 14:09:53.760195 16002 tablet_replica.cc:330] T ba183686332c4ab5b606bbc5f1b4e787 P a278448771e14471ab4df1e6649485bd: stopping tablet replica
I20240501 14:09:53.760414 16002 raft_consensus.cc:2229] T ba183686332c4ab5b606bbc5f1b4e787 P a278448771e14471ab4df1e6649485bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:53.760596 16002 raft_consensus.cc:2258] T ba183686332c4ab5b606bbc5f1b4e787 P a278448771e14471ab4df1e6649485bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:53.763032 16002 ts_tablet_manager.cc:1875] T ba183686332c4ab5b606bbc5f1b4e787 P a278448771e14471ab4df1e6649485bd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:53.779353 16002 ts_tablet_manager.cc:1888] T ba183686332c4ab5b606bbc5f1b4e787 P a278448771e14471ab4df1e6649485bd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.205
I20240501 14:09:53.784329 16002 log.cc:1196] T ba183686332c4ab5b606bbc5f1b4e787 P a278448771e14471ab4df1e6649485bd: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal/wals/ba183686332c4ab5b606bbc5f1b4e787
I20240501 14:09:53.785857 13288 catalog_manager.cc:4830] TS a278448771e14471ab4df1e6649485bd (127.7.39.131:46461): tablet ba183686332c4ab5b606bbc5f1b4e787 (table rebalance_test_table_2 [id=c8e8153727f143bdae4e267482cc7660]) successfully deleted
I20240501 14:09:54.140892 14665 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:54.143204 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:54.153378 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:54.157361 13555 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:54.161689 13687 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:54.234622 15062 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:54.245790 14928 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:54.443578 13576 consensus_queue.cc:237] T e4266d76760c4f84b16991f8aa38a3c3 P 43736346b1ff48c78ac919c80ba58654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 180, Committed index: 180, Last appended: 1.180, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 181 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } }
I20240501 14:09:54.447860 15081 raft_consensus.cc:1261] T e4266d76760c4f84b16991f8aa38a3c3 P 39bde7c058154f8aa40085db77f8c6e5 [term 1 FOLLOWER]: Refusing update from remote peer 43736346b1ff48c78ac919c80ba58654: Log matching property violated. Preceding OpId in replica: term: 1 index: 180. Preceding OpId from leader: term: 1 index: 181. (index mismatch)
I20240501 14:09:54.448945 15212 consensus_queue.cc:1035] T e4266d76760c4f84b16991f8aa38a3c3 P 43736346b1ff48c78ac919c80ba58654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 181, Last known committed idx: 180, Time since last communication: 0.000s
I20240501 14:09:54.455487 15782 raft_consensus.cc:2932] T e4266d76760c4f84b16991f8aa38a3c3 P 43736346b1ff48c78ac919c80ba58654 [term 1 LEADER]: Committing config change with OpId 1.181: config changed from index 45 to 181, VOTER 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129) evicted. New config: { opid_index: 181 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } } }
I20240501 14:09:54.459586 15080 raft_consensus.cc:2932] T e4266d76760c4f84b16991f8aa38a3c3 P 39bde7c058154f8aa40085db77f8c6e5 [term 1 FOLLOWER]: Committing config change with OpId 1.181: config changed from index 45 to 181, VOTER 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129) evicted. New config: { opid_index: 181 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } } }
I20240501 14:09:54.471550 13302 catalog_manager.cc:5484] T e4266d76760c4f84b16991f8aa38a3c3 P 43736346b1ff48c78ac919c80ba58654 reported cstate change: config changed from index 45 to 181, VOTER 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129) evicted. New cstate: current_term: 1 leader_uuid: "43736346b1ff48c78ac919c80ba58654" committed_config { opid_index: 181 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } } }
I20240501 14:09:54.531075 13426 tablet_service.cc:1514] Processing DeleteTablet for tablet e4266d76760c4f84b16991f8aa38a3c3 with delete_type TABLET_DATA_TOMBSTONED (TS 50d514f034204fc38331aa0a4fd74a1b not found in new config with opid_index 181) from {username='slave'} at 127.0.0.1:45262
I20240501 14:09:54.531697 16075 tablet_replica.cc:330] T e4266d76760c4f84b16991f8aa38a3c3 P 50d514f034204fc38331aa0a4fd74a1b: stopping tablet replica
I20240501 14:09:54.532042 16075 raft_consensus.cc:2229] T e4266d76760c4f84b16991f8aa38a3c3 P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:54.532227 16075 raft_consensus.cc:2258] T e4266d76760c4f84b16991f8aa38a3c3 P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:54.533023 16075 ts_tablet_manager.cc:1875] T e4266d76760c4f84b16991f8aa38a3c3 P 50d514f034204fc38331aa0a4fd74a1b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:54.554051 16075 ts_tablet_manager.cc:1888] T e4266d76760c4f84b16991f8aa38a3c3 P 50d514f034204fc38331aa0a4fd74a1b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.180
I20240501 14:09:54.554251 16075 log.cc:1196] T e4266d76760c4f84b16991f8aa38a3c3 P 50d514f034204fc38331aa0a4fd74a1b: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal/wals/e4266d76760c4f84b16991f8aa38a3c3
I20240501 14:09:54.554912 13291 catalog_manager.cc:4830] TS 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129:37933): tablet e4266d76760c4f84b16991f8aa38a3c3 (table rebalance_test_table_0 [id=b91b950953be4c81b3156e9c2c4e50ee]) successfully deleted
I20240501 14:09:54.915804 14665 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:54.920202 15062 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:54.924495 13555 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:54.947333 13687 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:54.967932 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:55.010267 14928 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:55.043424 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:55.337320 13707 consensus_queue.cc:237] T d948c2d5182b415b8aedb4c7b786247f P a278448771e14471ab4df1e6649485bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 285, Committed index: 285, Last appended: 1.285, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } }
I20240501 14:09:55.340755 14947 raft_consensus.cc:1261] T d948c2d5182b415b8aedb4c7b786247f P e26df472efe547f8bc5951dce352471b [term 1 FOLLOWER]: Refusing update from remote peer a278448771e14471ab4df1e6649485bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 285. Preceding OpId from leader: term: 1 index: 286. (index mismatch)
I20240501 14:09:55.341672 15938 consensus_queue.cc:1035] T d948c2d5182b415b8aedb4c7b786247f P a278448771e14471ab4df1e6649485bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 286, Last known committed idx: 285, Time since last communication: 0.000s
I20240501 14:09:55.344988 15938 raft_consensus.cc:2932] T d948c2d5182b415b8aedb4c7b786247f P a278448771e14471ab4df1e6649485bd [term 1 LEADER]: Committing config change with OpId 1.286: config changed from index 79 to 286, VOTER 43736346b1ff48c78ac919c80ba58654 (127.7.39.130) evicted. New config: { opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } }
I20240501 14:09:55.348034 13302 catalog_manager.cc:5484] T d948c2d5182b415b8aedb4c7b786247f P a278448771e14471ab4df1e6649485bd reported cstate change: config changed from index 79 to 286, VOTER 43736346b1ff48c78ac919c80ba58654 (127.7.39.130) evicted. New cstate: current_term: 1 leader_uuid: "a278448771e14471ab4df1e6649485bd" committed_config { opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } }
I20240501 14:09:55.350111 14947 raft_consensus.cc:2932] T d948c2d5182b415b8aedb4c7b786247f P e26df472efe547f8bc5951dce352471b [term 1 FOLLOWER]: Committing config change with OpId 1.286: config changed from index 79 to 286, VOTER 43736346b1ff48c78ac919c80ba58654 (127.7.39.130) evicted. New config: { opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } }
I20240501 14:09:55.383184 13555 tablet_service.cc:1514] Processing DeleteTablet for tablet d948c2d5182b415b8aedb4c7b786247f with delete_type TABLET_DATA_TOMBSTONED (TS 43736346b1ff48c78ac919c80ba58654 not found in new config with opid_index 286) from {username='slave'} at 127.0.0.1:57610
I20240501 14:09:55.413959 16143 tablet_replica.cc:330] T d948c2d5182b415b8aedb4c7b786247f P 43736346b1ff48c78ac919c80ba58654: stopping tablet replica
I20240501 14:09:55.416448 16143 raft_consensus.cc:2229] T d948c2d5182b415b8aedb4c7b786247f P 43736346b1ff48c78ac919c80ba58654 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:55.416843 16143 raft_consensus.cc:2258] T d948c2d5182b415b8aedb4c7b786247f P 43736346b1ff48c78ac919c80ba58654 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:55.434298 16143 ts_tablet_manager.cc:1875] T d948c2d5182b415b8aedb4c7b786247f P 43736346b1ff48c78ac919c80ba58654: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:55.445562 16143 ts_tablet_manager.cc:1888] T d948c2d5182b415b8aedb4c7b786247f P 43736346b1ff48c78ac919c80ba58654: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.285
I20240501 14:09:55.445785 16143 log.cc:1196] T d948c2d5182b415b8aedb4c7b786247f P 43736346b1ff48c78ac919c80ba58654: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal/wals/d948c2d5182b415b8aedb4c7b786247f
I20240501 14:09:55.452443 13291 catalog_manager.cc:4830] TS 43736346b1ff48c78ac919c80ba58654 (127.7.39.130:43311): tablet d948c2d5182b415b8aedb4c7b786247f (table rebalance_test_table_1 [id=b18f113d94ba4379ab99c6c63917bc1a]) successfully deleted
I20240501 14:09:55.786846 15062 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 14:09:55.791107 13643 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.39.132:43977, user_credentials={real_user=slave}} blocked reactor thread for 85441.7us
I20240501 14:09:55.798933 14665 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:55.814613 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:55.824998 13555 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:55.847501 14928 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:55.877184 13687 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:55.898183 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:55.990208 13577 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cdb343c0875e4709b9637c241dd4d582"
dest_uuid: "43736346b1ff48c78ac919c80ba58654"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52970
I20240501 14:09:55.990401 13577 raft_consensus.cc:595] T cdb343c0875e4709b9637c241dd4d582 P 43736346b1ff48c78ac919c80ba58654 [term 1 LEADER]: Received request to transfer leadership
W20240501 14:09:55.993368 13536 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60886: Service unavailable: leader transfer in progress
W20240501 14:09:56.000845 13536 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60886: Service unavailable: leader transfer in progress
W20240501 14:09:56.017374 13536 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60886: Service unavailable: leader transfer in progress
W20240501 14:09:56.022570 13536 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60886: Service unavailable: leader transfer in progress
W20240501 14:09:56.043748 13536 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60886: Service unavailable: leader transfer in progress
W20240501 14:09:56.058677 13536 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60886: Service unavailable: leader transfer in progress
W20240501 14:09:56.080937 13536 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60886: Service unavailable: leader transfer in progress
W20240501 14:09:56.092473 13536 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60886: Service unavailable: leader transfer in progress
W20240501 14:09:56.108426 13536 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60886: Service unavailable: leader transfer in progress
W20240501 14:09:56.126636 13535 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60886: Service unavailable: leader transfer in progress
W20240501 14:09:56.143424 13536 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60886: Service unavailable: leader transfer in progress
W20240501 14:09:56.158008 13536 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60886: Service unavailable: leader transfer in progress
W20240501 14:09:56.174304 13536 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60886: Service unavailable: leader transfer in progress
W20240501 14:09:56.191010 13536 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60886: Service unavailable: leader transfer in progress
W20240501 14:09:56.211980 13535 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60886: Service unavailable: leader transfer in progress
W20240501 14:09:56.229825 13535 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60886: Service unavailable: leader transfer in progress
I20240501 14:09:56.242944 15062 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 14:09:56.248930 13535 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60886: Service unavailable: leader transfer in progress
I20240501 14:09:56.249500 13555 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:56.262792 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 14:09:56.268311 13535 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60886: Service unavailable: leader transfer in progress
I20240501 14:09:56.288928 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 14:09:56.289657 13535 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60886: Service unavailable: leader transfer in progress
I20240501 14:09:56.311014 14928 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:56.338794 15782 raft_consensus.cc:979] T cdb343c0875e4709b9637c241dd4d582 P 43736346b1ff48c78ac919c80ba58654: : Instructing follower 39bde7c058154f8aa40085db77f8c6e5 to start an election
I20240501 14:09:56.338919 15782 raft_consensus.cc:1067] T cdb343c0875e4709b9637c241dd4d582 P 43736346b1ff48c78ac919c80ba58654 [term 1 LEADER]: Signalling peer 39bde7c058154f8aa40085db77f8c6e5 to start an election
W20240501 14:09:56.339605 13535 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60886: Service unavailable: leader transfer in progress
I20240501 14:09:56.340695 15081 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cdb343c0875e4709b9637c241dd4d582"
dest_uuid: "39bde7c058154f8aa40085db77f8c6e5"
from {username='slave'} at 127.7.39.130:60211
I20240501 14:09:56.340911 15081 raft_consensus.cc:482] T cdb343c0875e4709b9637c241dd4d582 P 39bde7c058154f8aa40085db77f8c6e5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:56.341029 15081 raft_consensus.cc:3037] T cdb343c0875e4709b9637c241dd4d582 P 39bde7c058154f8aa40085db77f8c6e5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:56.342126 15081 raft_consensus.cc:504] T cdb343c0875e4709b9637c241dd4d582 P 39bde7c058154f8aa40085db77f8c6e5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } }
I20240501 14:09:56.343794 14665 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:56.344204 15081 leader_election.cc:288] T cdb343c0875e4709b9637c241dd4d582 P 39bde7c058154f8aa40085db77f8c6e5 [CANDIDATE]: Term 2 election: Requested vote from peers 43736346b1ff48c78ac919c80ba58654 (127.7.39.130:43311), a278448771e14471ab4df1e6649485bd (127.7.39.131:46461)
I20240501 14:09:56.345269 13576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdb343c0875e4709b9637c241dd4d582" candidate_uuid: "39bde7c058154f8aa40085db77f8c6e5" candidate_term: 2 candidate_status { last_received { term: 1 index: 254 } } ignore_live_leader: true dest_uuid: "43736346b1ff48c78ac919c80ba58654"
I20240501 14:09:56.345386 13576 raft_consensus.cc:3032] T cdb343c0875e4709b9637c241dd4d582 P 43736346b1ff48c78ac919c80ba58654 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:09:56.345431 13576 raft_consensus.cc:729] T cdb343c0875e4709b9637c241dd4d582 P 43736346b1ff48c78ac919c80ba58654 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 43736346b1ff48c78ac919c80ba58654, State: Running, Role: LEADER
I20240501 14:09:56.345574 13576 consensus_queue.cc:260] T cdb343c0875e4709b9637c241dd4d582 P 43736346b1ff48c78ac919c80ba58654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 254, Committed index: 254, Last appended: 1.254, Last appended by leader: 254, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } }
I20240501 14:09:56.345739 13576 raft_consensus.cc:3037] T cdb343c0875e4709b9637c241dd4d582 P 43736346b1ff48c78ac919c80ba58654 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:56.346813 13576 raft_consensus.cc:2454] T cdb343c0875e4709b9637c241dd4d582 P 43736346b1ff48c78ac919c80ba58654 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39bde7c058154f8aa40085db77f8c6e5 in term 2.
I20240501 14:09:56.347263 15018 leader_election.cc:302] T cdb343c0875e4709b9637c241dd4d582 P 39bde7c058154f8aa40085db77f8c6e5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39bde7c058154f8aa40085db77f8c6e5, 43736346b1ff48c78ac919c80ba58654; no voters:
I20240501 14:09:56.371738 13687 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:09:56.376317 13536 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60886: Illegal state: replica 43736346b1ff48c78ac919c80ba58654 is not leader of this config: current role FOLLOWER
I20240501 14:09:56.400538 13707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdb343c0875e4709b9637c241dd4d582" candidate_uuid: "39bde7c058154f8aa40085db77f8c6e5" candidate_term: 2 candidate_status { last_received { term: 1 index: 254 } } ignore_live_leader: true dest_uuid: "a278448771e14471ab4df1e6649485bd"
I20240501 14:09:56.400691 13707 raft_consensus.cc:3037] T cdb343c0875e4709b9637c241dd4d582 P a278448771e14471ab4df1e6649485bd [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:56.401965 13707 raft_consensus.cc:2454] T cdb343c0875e4709b9637c241dd4d582 P a278448771e14471ab4df1e6649485bd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39bde7c058154f8aa40085db77f8c6e5 in term 2.
I20240501 14:09:56.403242 16228 raft_consensus.cc:2781] T cdb343c0875e4709b9637c241dd4d582 P 39bde7c058154f8aa40085db77f8c6e5 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:56.403623 16228 raft_consensus.cc:686] T cdb343c0875e4709b9637c241dd4d582 P 39bde7c058154f8aa40085db77f8c6e5 [term 2 LEADER]: Becoming Leader. State: Replica: 39bde7c058154f8aa40085db77f8c6e5, State: Running, Role: LEADER
I20240501 14:09:56.435266 16228 consensus_queue.cc:237] T cdb343c0875e4709b9637c241dd4d582 P 39bde7c058154f8aa40085db77f8c6e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 254, Committed index: 254, Last appended: 1.254, Last appended by leader: 254, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } }
I20240501 14:09:56.462141 13295 catalog_manager.cc:5484] T cdb343c0875e4709b9637c241dd4d582 P 39bde7c058154f8aa40085db77f8c6e5 reported cstate change: term changed from 1 to 2, leader changed from 43736346b1ff48c78ac919c80ba58654 (127.7.39.130) to 39bde7c058154f8aa40085db77f8c6e5 (127.7.39.135). New cstate: current_term: 2 leader_uuid: "39bde7c058154f8aa40085db77f8c6e5" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } } }
W20240501 14:09:56.491132 13666 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59038: Illegal state: replica a278448771e14471ab4df1e6649485bd is not leader of this config: current role FOLLOWER
I20240501 14:09:56.527396 13707 raft_consensus.cc:1261] T cdb343c0875e4709b9637c241dd4d582 P a278448771e14471ab4df1e6649485bd [term 2 FOLLOWER]: Refusing update from remote peer 39bde7c058154f8aa40085db77f8c6e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 254. Preceding OpId from leader: term: 2 index: 256. (index mismatch)
I20240501 14:09:56.528259 16228 consensus_queue.cc:1035] T cdb343c0875e4709b9637c241dd4d582 P 39bde7c058154f8aa40085db77f8c6e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 255, Last known committed idx: 254, Time since last communication: 0.000s
I20240501 14:09:56.539516 13576 raft_consensus.cc:1261] T cdb343c0875e4709b9637c241dd4d582 P 43736346b1ff48c78ac919c80ba58654 [term 2 FOLLOWER]: Refusing update from remote peer 39bde7c058154f8aa40085db77f8c6e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 254. Preceding OpId from leader: term: 2 index: 256. (index mismatch)
I20240501 14:09:56.541612 16228 consensus_queue.cc:1035] T cdb343c0875e4709b9637c241dd4d582 P 39bde7c058154f8aa40085db77f8c6e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 255, Last known committed idx: 254, Time since last communication: 0.000s
I20240501 14:09:56.803499 15081 consensus_queue.cc:237] T cdb343c0875e4709b9637c241dd4d582 P 39bde7c058154f8aa40085db77f8c6e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 264, Committed index: 264, Last appended: 2.264, Last appended by leader: 254, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 265 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } }
I20240501 14:09:56.805035 13706 raft_consensus.cc:1261] T cdb343c0875e4709b9637c241dd4d582 P a278448771e14471ab4df1e6649485bd [term 2 FOLLOWER]: Refusing update from remote peer 39bde7c058154f8aa40085db77f8c6e5: Log matching property violated. Preceding OpId in replica: term: 2 index: 264. Preceding OpId from leader: term: 2 index: 265. (index mismatch)
I20240501 14:09:56.805510 16228 consensus_queue.cc:1035] T cdb343c0875e4709b9637c241dd4d582 P 39bde7c058154f8aa40085db77f8c6e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 265, Last known committed idx: 264, Time since last communication: 0.000s
I20240501 14:09:56.807039 16228 raft_consensus.cc:2932] T cdb343c0875e4709b9637c241dd4d582 P 39bde7c058154f8aa40085db77f8c6e5 [term 2 LEADER]: Committing config change with OpId 2.265: config changed from index 26 to 265, VOTER 43736346b1ff48c78ac919c80ba58654 (127.7.39.130) evicted. New config: { opid_index: 265 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } } }
I20240501 14:09:56.809581 13706 raft_consensus.cc:2932] T cdb343c0875e4709b9637c241dd4d582 P a278448771e14471ab4df1e6649485bd [term 2 FOLLOWER]: Committing config change with OpId 2.265: config changed from index 26 to 265, VOTER 43736346b1ff48c78ac919c80ba58654 (127.7.39.130) evicted. New config: { opid_index: 265 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } } }
I20240501 14:09:56.812206 13302 catalog_manager.cc:5484] T cdb343c0875e4709b9637c241dd4d582 P a278448771e14471ab4df1e6649485bd reported cstate change: config changed from index 26 to 265, VOTER 43736346b1ff48c78ac919c80ba58654 (127.7.39.130) evicted. New cstate: current_term: 2 leader_uuid: "39bde7c058154f8aa40085db77f8c6e5" committed_config { opid_index: 265 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } } }
I20240501 14:09:56.815661 13555 tablet_service.cc:1514] Processing DeleteTablet for tablet cdb343c0875e4709b9637c241dd4d582 with delete_type TABLET_DATA_TOMBSTONED (TS 43736346b1ff48c78ac919c80ba58654 not found in new config with opid_index 265) from {username='slave'} at 127.0.0.1:57610
I20240501 14:09:56.844914 16264 tablet_replica.cc:330] T cdb343c0875e4709b9637c241dd4d582 P 43736346b1ff48c78ac919c80ba58654: stopping tablet replica
I20240501 14:09:56.845089 16264 raft_consensus.cc:2229] T cdb343c0875e4709b9637c241dd4d582 P 43736346b1ff48c78ac919c80ba58654 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:56.845227 16264 raft_consensus.cc:2258] T cdb343c0875e4709b9637c241dd4d582 P 43736346b1ff48c78ac919c80ba58654 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:56.849715 16264 ts_tablet_manager.cc:1875] T cdb343c0875e4709b9637c241dd4d582 P 43736346b1ff48c78ac919c80ba58654: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:56.852435 16264 ts_tablet_manager.cc:1888] T cdb343c0875e4709b9637c241dd4d582 P 43736346b1ff48c78ac919c80ba58654: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.264
I20240501 14:09:56.852588 16264 log.cc:1196] T cdb343c0875e4709b9637c241dd4d582 P 43736346b1ff48c78ac919c80ba58654: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal/wals/cdb343c0875e4709b9637c241dd4d582
I20240501 14:09:56.860924 13291 catalog_manager.cc:4830] TS 43736346b1ff48c78ac919c80ba58654 (127.7.39.130:43311): tablet cdb343c0875e4709b9637c241dd4d582 (table rebalance_test_table_3 [id=50bc5b382cee4078bfb9cc86c185c1b2]) successfully deleted
I20240501 14:09:57.138307 15062 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:57.229204 13687 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:57.242627 13555 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:57.248319 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:57.255743 14665 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:57.264295 14928 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:57.286477 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:57.506812 13706 consensus_queue.cc:237] T 4d3b3213e1dd426b8efc3fdc733b1842 P a278448771e14471ab4df1e6649485bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 335, Committed index: 335, Last appended: 1.335, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 336 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "2496e4af45ba458eacaacc9d25dcf01a" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 43977 } }
I20240501 14:09:57.508226 14684 raft_consensus.cc:1261] T 4d3b3213e1dd426b8efc3fdc733b1842 P 2496e4af45ba458eacaacc9d25dcf01a [term 1 FOLLOWER]: Refusing update from remote peer a278448771e14471ab4df1e6649485bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 335. Preceding OpId from leader: term: 1 index: 336. (index mismatch)
I20240501 14:09:57.509914 15938 consensus_queue.cc:1035] T 4d3b3213e1dd426b8efc3fdc733b1842 P a278448771e14471ab4df1e6649485bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "2496e4af45ba458eacaacc9d25dcf01a" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 336, Last known committed idx: 335, Time since last communication: 0.000s
I20240501 14:09:57.511984 14684 raft_consensus.cc:2932] T 4d3b3213e1dd426b8efc3fdc733b1842 P 2496e4af45ba458eacaacc9d25dcf01a [term 1 FOLLOWER]: Committing config change with OpId 1.336: config changed from index 80 to 336, VOTER 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129) evicted. New config: { opid_index: 336 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "2496e4af45ba458eacaacc9d25dcf01a" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 43977 } } }
I20240501 14:09:57.513906 15938 raft_consensus.cc:2932] T 4d3b3213e1dd426b8efc3fdc733b1842 P a278448771e14471ab4df1e6649485bd [term 1 LEADER]: Committing config change with OpId 1.336: config changed from index 80 to 336, VOTER 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129) evicted. New config: { opid_index: 336 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "2496e4af45ba458eacaacc9d25dcf01a" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 43977 } } }
I20240501 14:09:57.517180 13295 catalog_manager.cc:5484] T 4d3b3213e1dd426b8efc3fdc733b1842 P 2496e4af45ba458eacaacc9d25dcf01a reported cstate change: config changed from index 80 to 336, VOTER 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129) evicted. New cstate: current_term: 1 leader_uuid: "a278448771e14471ab4df1e6649485bd" committed_config { opid_index: 336 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "2496e4af45ba458eacaacc9d25dcf01a" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 43977 } } }
I20240501 14:09:57.542037 13426 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d3b3213e1dd426b8efc3fdc733b1842 with delete_type TABLET_DATA_TOMBSTONED (TS 50d514f034204fc38331aa0a4fd74a1b not found in new config with opid_index 336) from {username='slave'} at 127.0.0.1:45262
I20240501 14:09:57.571489 16339 tablet_replica.cc:330] T 4d3b3213e1dd426b8efc3fdc733b1842 P 50d514f034204fc38331aa0a4fd74a1b: stopping tablet replica
I20240501 14:09:57.572039 16339 raft_consensus.cc:2229] T 4d3b3213e1dd426b8efc3fdc733b1842 P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:57.572229 16339 raft_consensus.cc:2258] T 4d3b3213e1dd426b8efc3fdc733b1842 P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:57.578022 16339 ts_tablet_manager.cc:1875] T 4d3b3213e1dd426b8efc3fdc733b1842 P 50d514f034204fc38331aa0a4fd74a1b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:57.598017 16339 ts_tablet_manager.cc:1888] T 4d3b3213e1dd426b8efc3fdc733b1842 P 50d514f034204fc38331aa0a4fd74a1b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.335
I20240501 14:09:57.598215 16339 log.cc:1196] T 4d3b3213e1dd426b8efc3fdc733b1842 P 50d514f034204fc38331aa0a4fd74a1b: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal/wals/4d3b3213e1dd426b8efc3fdc733b1842
I20240501 14:09:57.600364 13291 catalog_manager.cc:4830] TS 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129:37933): tablet 4d3b3213e1dd426b8efc3fdc733b1842 (table rebalance_test_table_2 [id=c8e8153727f143bdae4e267482cc7660]) successfully deleted
I20240501 14:09:57.998386 13555 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:58.049350 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:58.056193 14665 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:58.057065 13687 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:58.069710 15062 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:58.080720 14928 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:58.080834 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:58.349951 13577 consensus_queue.cc:237] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 318, Committed index: 318, Last appended: 1.318, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } }
I20240501 14:09:58.351063 14815 raft_consensus.cc:1261] T bf0b5fd770b8405b9049c52d5d95bc53 P 4015e9df09424332bc3d7a9bad781641 [term 1 FOLLOWER]: Refusing update from remote peer 43736346b1ff48c78ac919c80ba58654: Log matching property violated. Preceding OpId in replica: term: 1 index: 318. Preceding OpId from leader: term: 1 index: 319. (index mismatch)
I20240501 14:09:58.358764 15782 consensus_queue.cc:1035] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 319, Last known committed idx: 318, Time since last communication: 0.000s
I20240501 14:09:58.363478 15568 raft_consensus.cc:2932] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654 [term 1 LEADER]: Committing config change with OpId 1.319: config changed from index 35 to 319, VOTER a278448771e14471ab4df1e6649485bd (127.7.39.131) evicted. New config: { opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } }
I20240501 14:09:58.363857 14816 raft_consensus.cc:2932] T bf0b5fd770b8405b9049c52d5d95bc53 P 4015e9df09424332bc3d7a9bad781641 [term 1 FOLLOWER]: Committing config change with OpId 1.319: config changed from index 35 to 319, VOTER a278448771e14471ab4df1e6649485bd (127.7.39.131) evicted. New config: { opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } }
I20240501 14:09:58.366650 13301 catalog_manager.cc:5484] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654 reported cstate change: config changed from index 35 to 319, VOTER a278448771e14471ab4df1e6649485bd (127.7.39.131) evicted. New cstate: current_term: 1 leader_uuid: "43736346b1ff48c78ac919c80ba58654" committed_config { opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } }
I20240501 14:09:58.445370 13687 tablet_service.cc:1514] Processing DeleteTablet for tablet bf0b5fd770b8405b9049c52d5d95bc53 with delete_type TABLET_DATA_TOMBSTONED (TS a278448771e14471ab4df1e6649485bd not found in new config with opid_index 319) from {username='slave'} at 127.0.0.1:45074
I20240501 14:09:58.462230 16416 tablet_replica.cc:330] T bf0b5fd770b8405b9049c52d5d95bc53 P a278448771e14471ab4df1e6649485bd: stopping tablet replica
I20240501 14:09:58.462479 16416 raft_consensus.cc:2229] T bf0b5fd770b8405b9049c52d5d95bc53 P a278448771e14471ab4df1e6649485bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:58.463140 16416 raft_consensus.cc:2258] T bf0b5fd770b8405b9049c52d5d95bc53 P a278448771e14471ab4df1e6649485bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:58.464129 16416 ts_tablet_manager.cc:1875] T bf0b5fd770b8405b9049c52d5d95bc53 P a278448771e14471ab4df1e6649485bd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:58.471410 16416 ts_tablet_manager.cc:1888] T bf0b5fd770b8405b9049c52d5d95bc53 P a278448771e14471ab4df1e6649485bd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.318
I20240501 14:09:58.471567 16416 log.cc:1196] T bf0b5fd770b8405b9049c52d5d95bc53 P a278448771e14471ab4df1e6649485bd: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal/wals/bf0b5fd770b8405b9049c52d5d95bc53
I20240501 14:09:58.472368 13288 catalog_manager.cc:4830] TS a278448771e14471ab4df1e6649485bd (127.7.39.131:46461): tablet bf0b5fd770b8405b9049c52d5d95bc53 (table rebalance_test_table_3 [id=50bc5b382cee4078bfb9cc86c185c1b2]) successfully deleted
I20240501 14:09:58.718876 13555 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:58.721446 14665 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:58.725272 13687 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:58.730909 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:58.742558 14928 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:58.788336 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:58.808961 15062 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:59.057334 13707 consensus_queue.cc:237] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 294, Committed index: 294, Last appended: 1.294, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } }
I20240501 14:09:59.061558 14815 raft_consensus.cc:1261] T a067a9d6642349c1b99effece908888c P 4015e9df09424332bc3d7a9bad781641 [term 1 FOLLOWER]: Refusing update from remote peer a278448771e14471ab4df1e6649485bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 294. Preceding OpId from leader: term: 1 index: 295. (index mismatch)
I20240501 14:09:59.061940 15937 consensus_queue.cc:1035] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 295, Last known committed idx: 294, Time since last communication: 0.000s
I20240501 14:09:59.068202 15537 raft_consensus.cc:2932] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd [term 1 LEADER]: Committing config change with OpId 1.295: config changed from index 27 to 295, VOTER 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129) evicted. New config: { opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } }
I20240501 14:09:59.081188 14815 raft_consensus.cc:2932] T a067a9d6642349c1b99effece908888c P 4015e9df09424332bc3d7a9bad781641 [term 1 FOLLOWER]: Committing config change with OpId 1.295: config changed from index 27 to 295, VOTER 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129) evicted. New config: { opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } }
I20240501 14:09:59.081955 13301 catalog_manager.cc:5484] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd reported cstate change: config changed from index 27 to 295, VOTER 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129) evicted. New cstate: current_term: 1 leader_uuid: "a278448771e14471ab4df1e6649485bd" committed_config { opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } }
I20240501 14:09:59.109953 13426 tablet_service.cc:1514] Processing DeleteTablet for tablet a067a9d6642349c1b99effece908888c with delete_type TABLET_DATA_TOMBSTONED (TS 50d514f034204fc38331aa0a4fd74a1b not found in new config with opid_index 295) from {username='slave'} at 127.0.0.1:45262
I20240501 14:09:59.120239 16479 tablet_replica.cc:330] T a067a9d6642349c1b99effece908888c P 50d514f034204fc38331aa0a4fd74a1b: stopping tablet replica
I20240501 14:09:59.120822 16479 raft_consensus.cc:2229] T a067a9d6642349c1b99effece908888c P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:59.121011 16479 raft_consensus.cc:2258] T a067a9d6642349c1b99effece908888c P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:59.121870 16479 ts_tablet_manager.cc:1875] T a067a9d6642349c1b99effece908888c P 50d514f034204fc38331aa0a4fd74a1b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:59.138830 16479 ts_tablet_manager.cc:1888] T a067a9d6642349c1b99effece908888c P 50d514f034204fc38331aa0a4fd74a1b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.294
I20240501 14:09:59.139209 16479 log.cc:1196] T a067a9d6642349c1b99effece908888c P 50d514f034204fc38331aa0a4fd74a1b: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal/wals/a067a9d6642349c1b99effece908888c
I20240501 14:09:59.140242 13291 catalog_manager.cc:4830] TS 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129:37933): tablet a067a9d6642349c1b99effece908888c (table rebalance_test_table_4 [id=7319ca2094424c85b7008d8839665ed4]) successfully deleted
I20240501 14:09:59.357220 13555 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:59.361747 13687 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:59.363484 14665 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:59.377645 14928 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:59.390738 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:59.390769 15062 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:59.405794 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:59.652776 13707 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cde568b4e46a4c5499031d0e18ae51de"
dest_uuid: "a278448771e14471ab4df1e6649485bd"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59160
I20240501 14:09:59.655551 13707 raft_consensus.cc:595] T cde568b4e46a4c5499031d0e18ae51de P a278448771e14471ab4df1e6649485bd [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:59.901911 14665 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:59.911850 13555 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:59.920094 15062 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:59.927611 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:59.933660 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:59.943940 13687 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:59.974238 14928 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:00.033516 15000 raft_consensus.cc:979] T cde568b4e46a4c5499031d0e18ae51de P a278448771e14471ab4df1e6649485bd: : Instructing follower 2496e4af45ba458eacaacc9d25dcf01a to start an election
I20240501 14:10:00.033645 15000 raft_consensus.cc:1067] T cde568b4e46a4c5499031d0e18ae51de P a278448771e14471ab4df1e6649485bd [term 1 LEADER]: Signalling peer 2496e4af45ba458eacaacc9d25dcf01a to start an election
I20240501 14:10:00.050747 14685 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cde568b4e46a4c5499031d0e18ae51de"
dest_uuid: "2496e4af45ba458eacaacc9d25dcf01a"
from {username='slave'} at 127.7.39.131:34959
I20240501 14:10:00.052819 14685 raft_consensus.cc:482] T cde568b4e46a4c5499031d0e18ae51de P 2496e4af45ba458eacaacc9d25dcf01a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:10:00.053424 14685 raft_consensus.cc:3037] T cde568b4e46a4c5499031d0e18ae51de P 2496e4af45ba458eacaacc9d25dcf01a [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:00.066953 14685 raft_consensus.cc:504] T cde568b4e46a4c5499031d0e18ae51de P 2496e4af45ba458eacaacc9d25dcf01a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "2496e4af45ba458eacaacc9d25dcf01a" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 43977 } }
I20240501 14:10:00.068473 14685 leader_election.cc:288] T cde568b4e46a4c5499031d0e18ae51de P 2496e4af45ba458eacaacc9d25dcf01a [CANDIDATE]: Term 2 election: Requested vote from peers 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129:37933), a278448771e14471ab4df1e6649485bd (127.7.39.131:46461)
I20240501 14:10:00.068789 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cde568b4e46a4c5499031d0e18ae51de" candidate_uuid: "2496e4af45ba458eacaacc9d25dcf01a" candidate_term: 2 candidate_status { last_received { term: 1 index: 311 } } ignore_live_leader: true dest_uuid: "a278448771e14471ab4df1e6649485bd"
I20240501 14:10:00.068822 13442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cde568b4e46a4c5499031d0e18ae51de" candidate_uuid: "2496e4af45ba458eacaacc9d25dcf01a" candidate_term: 2 candidate_status { last_received { term: 1 index: 311 } } ignore_live_leader: true dest_uuid: "50d514f034204fc38331aa0a4fd74a1b"
I20240501 14:10:00.068930 13708 raft_consensus.cc:3032] T cde568b4e46a4c5499031d0e18ae51de P a278448771e14471ab4df1e6649485bd [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:10:00.068953 13442 raft_consensus.cc:3037] T cde568b4e46a4c5499031d0e18ae51de P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:00.068977 13708 raft_consensus.cc:729] T cde568b4e46a4c5499031d0e18ae51de P a278448771e14471ab4df1e6649485bd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a278448771e14471ab4df1e6649485bd, State: Running, Role: LEADER
I20240501 14:10:00.069129 13708 consensus_queue.cc:260] T cde568b4e46a4c5499031d0e18ae51de P a278448771e14471ab4df1e6649485bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 311, Committed index: 311, Last appended: 1.311, Last appended by leader: 311, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "2496e4af45ba458eacaacc9d25dcf01a" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 43977 } }
I20240501 14:10:00.069280 13708 raft_consensus.cc:3037] T cde568b4e46a4c5499031d0e18ae51de P a278448771e14471ab4df1e6649485bd [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:00.070019 13442 raft_consensus.cc:2454] T cde568b4e46a4c5499031d0e18ae51de P 50d514f034204fc38331aa0a4fd74a1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2496e4af45ba458eacaacc9d25dcf01a in term 2.
I20240501 14:10:00.070380 13708 raft_consensus.cc:2454] T cde568b4e46a4c5499031d0e18ae51de P a278448771e14471ab4df1e6649485bd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2496e4af45ba458eacaacc9d25dcf01a in term 2.
I20240501 14:10:00.070442 14621 leader_election.cc:302] T cde568b4e46a4c5499031d0e18ae51de P 2496e4af45ba458eacaacc9d25dcf01a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2496e4af45ba458eacaacc9d25dcf01a, 50d514f034204fc38331aa0a4fd74a1b; no voters:
I20240501 14:10:00.092006 16579 raft_consensus.cc:2781] T cde568b4e46a4c5499031d0e18ae51de P 2496e4af45ba458eacaacc9d25dcf01a [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:10:00.092360 16579 raft_consensus.cc:686] T cde568b4e46a4c5499031d0e18ae51de P 2496e4af45ba458eacaacc9d25dcf01a [term 2 LEADER]: Becoming Leader. State: Replica: 2496e4af45ba458eacaacc9d25dcf01a, State: Running, Role: LEADER
I20240501 14:10:00.092885 16579 consensus_queue.cc:237] T cde568b4e46a4c5499031d0e18ae51de P 2496e4af45ba458eacaacc9d25dcf01a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 311, Committed index: 311, Last appended: 1.311, Last appended by leader: 311, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "2496e4af45ba458eacaacc9d25dcf01a" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 43977 } }
I20240501 14:10:00.097244 13302 catalog_manager.cc:5484] T cde568b4e46a4c5499031d0e18ae51de P 2496e4af45ba458eacaacc9d25dcf01a reported cstate change: term changed from 1 to 2, leader changed from a278448771e14471ab4df1e6649485bd (127.7.39.131) to 2496e4af45ba458eacaacc9d25dcf01a (127.7.39.132). New cstate: current_term: 2 leader_uuid: "2496e4af45ba458eacaacc9d25dcf01a" committed_config { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "2496e4af45ba458eacaacc9d25dcf01a" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 43977 } } }
W20240501 14:10:00.147665 13667 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45188: Illegal state: replica a278448771e14471ab4df1e6649485bd is not leader of this config: current role FOLLOWER
W20240501 14:10:00.209893 13406 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51808: Illegal state: replica 50d514f034204fc38331aa0a4fd74a1b is not leader of this config: current role FOLLOWER
I20240501 14:10:00.240943 13443 raft_consensus.cc:1261] T cde568b4e46a4c5499031d0e18ae51de P 50d514f034204fc38331aa0a4fd74a1b [term 2 FOLLOWER]: Refusing update from remote peer 2496e4af45ba458eacaacc9d25dcf01a: Log matching property violated. Preceding OpId in replica: term: 1 index: 311. Preceding OpId from leader: term: 2 index: 313. (index mismatch)
I20240501 14:10:00.241568 16579 consensus_queue.cc:1035] T cde568b4e46a4c5499031d0e18ae51de P 2496e4af45ba458eacaacc9d25dcf01a [LEADER]: Connected to new peer: Peer: permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 312, Last known committed idx: 311, Time since last communication: 0.000s
I20240501 14:10:00.243796 13708 raft_consensus.cc:1261] T cde568b4e46a4c5499031d0e18ae51de P a278448771e14471ab4df1e6649485bd [term 2 FOLLOWER]: Refusing update from remote peer 2496e4af45ba458eacaacc9d25dcf01a: Log matching property violated. Preceding OpId in replica: term: 1 index: 311. Preceding OpId from leader: term: 2 index: 313. (index mismatch)
I20240501 14:10:00.244920 16579 consensus_queue.cc:1035] T cde568b4e46a4c5499031d0e18ae51de P 2496e4af45ba458eacaacc9d25dcf01a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 312, Last known committed idx: 311, Time since last communication: 0.000s
I20240501 14:10:00.296586 15062 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:00.299896 14665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:00.306618 13555 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:00.306810 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:00.319958 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:00.337842 13687 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:00.361291 14928 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:00.664098 13446 consensus_queue.cc:237] T 341c6b416c374a9ba4fc34d861ba2ef9 P 50d514f034204fc38331aa0a4fd74a1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 339, Committed index: 339, Last appended: 1.340, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } }
I20240501 14:10:00.667454 14815 raft_consensus.cc:1261] T 341c6b416c374a9ba4fc34d861ba2ef9 P 4015e9df09424332bc3d7a9bad781641 [term 1 FOLLOWER]: Refusing update from remote peer 50d514f034204fc38331aa0a4fd74a1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 340. Preceding OpId from leader: term: 1 index: 341. (index mismatch)
I20240501 14:10:00.667917 15768 consensus_queue.cc:1035] T 341c6b416c374a9ba4fc34d861ba2ef9 P 50d514f034204fc38331aa0a4fd74a1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 341, Last known committed idx: 339, Time since last communication: 0.000s
I20240501 14:10:00.669468 15768 raft_consensus.cc:2932] T 341c6b416c374a9ba4fc34d861ba2ef9 P 50d514f034204fc38331aa0a4fd74a1b [term 1 LEADER]: Committing config change with OpId 1.341: config changed from index 65 to 341, VOTER a278448771e14471ab4df1e6649485bd (127.7.39.131) evicted. New config: { opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } }
I20240501 14:10:00.670240 14815 raft_consensus.cc:2932] T 341c6b416c374a9ba4fc34d861ba2ef9 P 4015e9df09424332bc3d7a9bad781641 [term 1 FOLLOWER]: Committing config change with OpId 1.341: config changed from index 65 to 341, VOTER a278448771e14471ab4df1e6649485bd (127.7.39.131) evicted. New config: { opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } }
I20240501 14:10:00.672983 13295 catalog_manager.cc:5484] T 341c6b416c374a9ba4fc34d861ba2ef9 P 50d514f034204fc38331aa0a4fd74a1b reported cstate change: config changed from index 65 to 341, VOTER a278448771e14471ab4df1e6649485bd (127.7.39.131) evicted. New cstate: current_term: 1 leader_uuid: "50d514f034204fc38331aa0a4fd74a1b" committed_config { opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } }
I20240501 14:10:00.677347 13687 tablet_service.cc:1514] Processing DeleteTablet for tablet 341c6b416c374a9ba4fc34d861ba2ef9 with delete_type TABLET_DATA_TOMBSTONED (TS a278448771e14471ab4df1e6649485bd not found in new config with opid_index 341) from {username='slave'} at 127.0.0.1:45074
I20240501 14:10:00.688515 16646 tablet_replica.cc:330] T 341c6b416c374a9ba4fc34d861ba2ef9 P a278448771e14471ab4df1e6649485bd: stopping tablet replica
I20240501 14:10:00.690783 16646 raft_consensus.cc:2229] T 341c6b416c374a9ba4fc34d861ba2ef9 P a278448771e14471ab4df1e6649485bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:00.690968 16646 pending_rounds.cc:62] T 341c6b416c374a9ba4fc34d861ba2ef9 P a278448771e14471ab4df1e6649485bd: Trying to abort 1 pending ops.
I20240501 14:10:00.691358 16646 pending_rounds.cc:68] T 341c6b416c374a9ba4fc34d861ba2ef9 P a278448771e14471ab4df1e6649485bd: Aborting op as it isn't in flight: id { term: 1 index: 340 } timestamp: 7022889372316409856 op_type: WRITE_OP write_request { tablet_id: "341c6b416c374a9ba4fc34d861ba2ef9" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\244\273Cj\005\037Y\213\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""8fccbbe79488c73e" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7022889372243689472 authz_token { token_data: "\010\326\234\311\261\006\"3\n\005slave\022*\n b91b950953be4c81b3156e9c2c4e50ee\020\001\030\001 \001(\001" signature: "<redacted>""lx;<)\217\021\263\256\250?\255^S\362\021\350J\251[\365\367\221\343]\333#\273\346\246M\036\331\237\351\225\343\345\037\212\256F\037\005\303\014-\236\023G\352V2OM\236r\027\330\247.&\217\313" signing_key_seq_num: 0 } } request_id { client_id: "20f101156dce4a419839f1279e26a939" seq_no: 1058 first_incomplete_seq_no: 1058 attempt_no: 0 }
I20240501 14:10:00.692387 16646 raft_consensus.cc:2258] T 341c6b416c374a9ba4fc34d861ba2ef9 P a278448771e14471ab4df1e6649485bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:00.693188 16646 ts_tablet_manager.cc:1875] T 341c6b416c374a9ba4fc34d861ba2ef9 P a278448771e14471ab4df1e6649485bd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:00.707983 16646 ts_tablet_manager.cc:1888] T 341c6b416c374a9ba4fc34d861ba2ef9 P a278448771e14471ab4df1e6649485bd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.340
I20240501 14:10:00.708194 16646 log.cc:1196] T 341c6b416c374a9ba4fc34d861ba2ef9 P a278448771e14471ab4df1e6649485bd: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal/wals/341c6b416c374a9ba4fc34d861ba2ef9
I20240501 14:10:00.709124 13288 catalog_manager.cc:4830] TS a278448771e14471ab4df1e6649485bd (127.7.39.131:46461): tablet 341c6b416c374a9ba4fc34d861ba2ef9 (table rebalance_test_table_0 [id=b91b950953be4c81b3156e9c2c4e50ee]) successfully deleted
I20240501 14:10:01.044270 13555 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:01.049139 15062 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:01.052492 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:01.066546 14665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:01.073468 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:01.089704 13687 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:01.102317 14928 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:01.355473 13576 consensus_queue.cc:237] T 0dc94f932e9a4319988e700aff042f22 P 43736346b1ff48c78ac919c80ba58654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 474, Committed index: 474, Last appended: 1.474, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 475 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } }
I20240501 14:10:01.361804 14947 raft_consensus.cc:1261] T 0dc94f932e9a4319988e700aff042f22 P e26df472efe547f8bc5951dce352471b [term 1 FOLLOWER]: Refusing update from remote peer 43736346b1ff48c78ac919c80ba58654: Log matching property violated. Preceding OpId in replica: term: 1 index: 474. Preceding OpId from leader: term: 1 index: 475. (index mismatch)
I20240501 14:10:01.363539 16450 consensus_queue.cc:1035] T 0dc94f932e9a4319988e700aff042f22 P 43736346b1ff48c78ac919c80ba58654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 475, Last known committed idx: 474, Time since last communication: 0.000s
I20240501 14:10:01.365677 16014 raft_consensus.cc:2932] T 0dc94f932e9a4319988e700aff042f22 P 43736346b1ff48c78ac919c80ba58654 [term 1 LEADER]: Committing config change with OpId 1.475: config changed from index 58 to 475, VOTER 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129) evicted. New config: { opid_index: 475 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } }
I20240501 14:10:01.366999 14947 raft_consensus.cc:2932] T 0dc94f932e9a4319988e700aff042f22 P e26df472efe547f8bc5951dce352471b [term 1 FOLLOWER]: Committing config change with OpId 1.475: config changed from index 58 to 475, VOTER 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129) evicted. New config: { opid_index: 475 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } }
I20240501 14:10:01.369366 13302 catalog_manager.cc:5484] T 0dc94f932e9a4319988e700aff042f22 P 43736346b1ff48c78ac919c80ba58654 reported cstate change: config changed from index 58 to 475, VOTER 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129) evicted. New cstate: current_term: 1 leader_uuid: "43736346b1ff48c78ac919c80ba58654" committed_config { opid_index: 475 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } }
I20240501 14:10:01.410003 13426 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dc94f932e9a4319988e700aff042f22 with delete_type TABLET_DATA_TOMBSTONED (TS 50d514f034204fc38331aa0a4fd74a1b not found in new config with opid_index 475) from {username='slave'} at 127.0.0.1:45262
I20240501 14:10:01.416993 16729 tablet_replica.cc:330] T 0dc94f932e9a4319988e700aff042f22 P 50d514f034204fc38331aa0a4fd74a1b: stopping tablet replica
I20240501 14:10:01.417208 16729 raft_consensus.cc:2229] T 0dc94f932e9a4319988e700aff042f22 P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:01.417374 16729 raft_consensus.cc:2258] T 0dc94f932e9a4319988e700aff042f22 P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:01.423202 16729 ts_tablet_manager.cc:1875] T 0dc94f932e9a4319988e700aff042f22 P 50d514f034204fc38331aa0a4fd74a1b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:01.459908 16729 ts_tablet_manager.cc:1888] T 0dc94f932e9a4319988e700aff042f22 P 50d514f034204fc38331aa0a4fd74a1b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.474
I20240501 14:10:01.460120 16729 log.cc:1196] T 0dc94f932e9a4319988e700aff042f22 P 50d514f034204fc38331aa0a4fd74a1b: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal/wals/0dc94f932e9a4319988e700aff042f22
I20240501 14:10:01.461136 13291 catalog_manager.cc:4830] TS 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129:37933): tablet 0dc94f932e9a4319988e700aff042f22 (table rebalance_test_table_2 [id=c8e8153727f143bdae4e267482cc7660]) successfully deleted
I20240501 14:10:01.653506 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:01.696801 14665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:01.701336 15062 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:01.723912 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:01.724690 13687 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:01.751853 13555 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:01.854983 14928 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:02.066255 13442 consensus_queue.cc:237] T 3f34112bba8a43a58420148533a6e989 P 50d514f034204fc38331aa0a4fd74a1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 520, Committed index: 520, Last appended: 1.520, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 521 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "2496e4af45ba458eacaacc9d25dcf01a" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 43977 } }
I20240501 14:10:02.067559 14685 raft_consensus.cc:1261] T 3f34112bba8a43a58420148533a6e989 P 2496e4af45ba458eacaacc9d25dcf01a [term 1 FOLLOWER]: Refusing update from remote peer 50d514f034204fc38331aa0a4fd74a1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 520. Preceding OpId from leader: term: 1 index: 521. (index mismatch)
I20240501 14:10:02.067955 15587 consensus_queue.cc:1035] T 3f34112bba8a43a58420148533a6e989 P 50d514f034204fc38331aa0a4fd74a1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2496e4af45ba458eacaacc9d25dcf01a" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 521, Last known committed idx: 520, Time since last communication: 0.000s
I20240501 14:10:02.070238 15587 raft_consensus.cc:2932] T 3f34112bba8a43a58420148533a6e989 P 50d514f034204fc38331aa0a4fd74a1b [term 1 LEADER]: Committing config change with OpId 1.521: config changed from index 81 to 521, VOTER 43736346b1ff48c78ac919c80ba58654 (127.7.39.130) evicted. New config: { opid_index: 521 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "2496e4af45ba458eacaacc9d25dcf01a" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 43977 } } }
I20240501 14:10:02.071533 14685 raft_consensus.cc:2932] T 3f34112bba8a43a58420148533a6e989 P 2496e4af45ba458eacaacc9d25dcf01a [term 1 FOLLOWER]: Committing config change with OpId 1.521: config changed from index 81 to 521, VOTER 43736346b1ff48c78ac919c80ba58654 (127.7.39.130) evicted. New config: { opid_index: 521 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "2496e4af45ba458eacaacc9d25dcf01a" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 43977 } } }
I20240501 14:10:02.076401 13295 catalog_manager.cc:5484] T 3f34112bba8a43a58420148533a6e989 P 2496e4af45ba458eacaacc9d25dcf01a reported cstate change: config changed from index 81 to 521, VOTER 43736346b1ff48c78ac919c80ba58654 (127.7.39.130) evicted. New cstate: current_term: 1 leader_uuid: "50d514f034204fc38331aa0a4fd74a1b" committed_config { opid_index: 521 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "2496e4af45ba458eacaacc9d25dcf01a" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 43977 } } }
I20240501 14:10:02.090723 13555 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f34112bba8a43a58420148533a6e989 with delete_type TABLET_DATA_TOMBSTONED (TS 43736346b1ff48c78ac919c80ba58654 not found in new config with opid_index 521) from {username='slave'} at 127.0.0.1:57610
I20240501 14:10:02.092931 16790 tablet_replica.cc:330] T 3f34112bba8a43a58420148533a6e989 P 43736346b1ff48c78ac919c80ba58654: stopping tablet replica
I20240501 14:10:02.093132 16790 raft_consensus.cc:2229] T 3f34112bba8a43a58420148533a6e989 P 43736346b1ff48c78ac919c80ba58654 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:02.093362 16790 raft_consensus.cc:2258] T 3f34112bba8a43a58420148533a6e989 P 43736346b1ff48c78ac919c80ba58654 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:02.111294 16790 ts_tablet_manager.cc:1875] T 3f34112bba8a43a58420148533a6e989 P 43736346b1ff48c78ac919c80ba58654: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:02.120256 16790 ts_tablet_manager.cc:1888] T 3f34112bba8a43a58420148533a6e989 P 43736346b1ff48c78ac919c80ba58654: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.520
I20240501 14:10:02.120400 16790 log.cc:1196] T 3f34112bba8a43a58420148533a6e989 P 43736346b1ff48c78ac919c80ba58654: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal/wals/3f34112bba8a43a58420148533a6e989
I20240501 14:10:02.121174 13291 catalog_manager.cc:4830] TS 43736346b1ff48c78ac919c80ba58654 (127.7.39.130:43311): tablet 3f34112bba8a43a58420148533a6e989 (table rebalance_test_table_1 [id=b18f113d94ba4379ab99c6c63917bc1a]) successfully deleted
I20240501 14:10:02.521602 13687 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:02.522842 15062 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:02.532454 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:02.533591 14665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:02.563031 13555 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:02.582381 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:02.617908 14928 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:02.717067 13577 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cce2260455ac4edbb9bc54b103c50eb3"
dest_uuid: "43736346b1ff48c78ac919c80ba58654"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47070
I20240501 14:10:02.717222 13577 raft_consensus.cc:595] T cce2260455ac4edbb9bc54b103c50eb3 P 43736346b1ff48c78ac919c80ba58654 [term 1 LEADER]: Received request to transfer leadership
W20240501 14:10:02.731786 13536 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60886: Service unavailable: leader transfer in progress
W20240501 14:10:02.739657 13536 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60886: Service unavailable: leader transfer in progress
W20240501 14:10:02.750020 13536 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60886: Service unavailable: leader transfer in progress
W20240501 14:10:02.758462 13536 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60886: Service unavailable: leader transfer in progress
W20240501 14:10:02.766232 13536 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60886: Service unavailable: leader transfer in progress
W20240501 14:10:02.775161 13534 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60886: Service unavailable: leader transfer in progress
W20240501 14:10:02.787928 13534 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60886: Service unavailable: leader transfer in progress
W20240501 14:10:02.799906 13534 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60886: Service unavailable: leader transfer in progress
W20240501 14:10:02.810957 13536 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60886: Service unavailable: leader transfer in progress
W20240501 14:10:02.824914 13534 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60886: Service unavailable: leader transfer in progress
W20240501 14:10:02.836742 13534 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60886: Service unavailable: leader transfer in progress
W20240501 14:10:02.849625 13534 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60886: Service unavailable: leader transfer in progress
W20240501 14:10:02.867295 13534 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60886: Service unavailable: leader transfer in progress
W20240501 14:10:02.884193 13534 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60886: Service unavailable: leader transfer in progress
I20240501 14:10:02.899133 14665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:10:02.902518 13534 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60886: Service unavailable: leader transfer in progress
I20240501 14:10:02.909579 15062 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:02.912535 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:02.918805 13555 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:02.925700 14928 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:02.927193 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 14:10:02.931074 13534 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60886: Service unavailable: leader transfer in progress
I20240501 14:10:02.941588 13687 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:10:02.955315 13536 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60886: Service unavailable: leader transfer in progress
W20240501 14:10:02.978497 13536 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60886: Service unavailable: leader transfer in progress
W20240501 14:10:02.998524 13536 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60886: Service unavailable: leader transfer in progress
W20240501 14:10:03.022243 13536 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60886: Service unavailable: leader transfer in progress
W20240501 14:10:03.045362 13534 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60886: Service unavailable: leader transfer in progress
W20240501 14:10:03.072705 13534 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60886: Service unavailable: leader transfer in progress
W20240501 14:10:03.098016 13534 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60886: Service unavailable: leader transfer in progress
W20240501 14:10:03.125479 13534 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60886: Service unavailable: leader transfer in progress
I20240501 14:10:03.130699 15568 raft_consensus.cc:979] T cce2260455ac4edbb9bc54b103c50eb3 P 43736346b1ff48c78ac919c80ba58654: : Instructing follower e26df472efe547f8bc5951dce352471b to start an election
I20240501 14:10:03.130824 15782 raft_consensus.cc:1067] T cce2260455ac4edbb9bc54b103c50eb3 P 43736346b1ff48c78ac919c80ba58654 [term 1 LEADER]: Signalling peer e26df472efe547f8bc5951dce352471b to start an election
I20240501 14:10:03.131259 14948 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cce2260455ac4edbb9bc54b103c50eb3"
dest_uuid: "e26df472efe547f8bc5951dce352471b"
from {username='slave'} at 127.7.39.130:39903
I20240501 14:10:03.131421 14948 raft_consensus.cc:482] T cce2260455ac4edbb9bc54b103c50eb3 P e26df472efe547f8bc5951dce352471b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:10:03.131486 14948 raft_consensus.cc:3037] T cce2260455ac4edbb9bc54b103c50eb3 P e26df472efe547f8bc5951dce352471b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:03.135818 14948 raft_consensus.cc:504] T cce2260455ac4edbb9bc54b103c50eb3 P e26df472efe547f8bc5951dce352471b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } }
I20240501 14:10:03.136435 14948 leader_election.cc:288] T cce2260455ac4edbb9bc54b103c50eb3 P e26df472efe547f8bc5951dce352471b [CANDIDATE]: Term 2 election: Requested vote from peers 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129:37933), 43736346b1ff48c78ac919c80ba58654 (127.7.39.130:43311)
I20240501 14:10:03.137287 13577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cce2260455ac4edbb9bc54b103c50eb3" candidate_uuid: "e26df472efe547f8bc5951dce352471b" candidate_term: 2 candidate_status { last_received { term: 1 index: 450 } } ignore_live_leader: true dest_uuid: "43736346b1ff48c78ac919c80ba58654"
I20240501 14:10:03.137408 13577 raft_consensus.cc:3032] T cce2260455ac4edbb9bc54b103c50eb3 P 43736346b1ff48c78ac919c80ba58654 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:10:03.137450 13577 raft_consensus.cc:729] T cce2260455ac4edbb9bc54b103c50eb3 P 43736346b1ff48c78ac919c80ba58654 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 43736346b1ff48c78ac919c80ba58654, State: Running, Role: LEADER
I20240501 14:10:03.137565 13577 consensus_queue.cc:260] T cce2260455ac4edbb9bc54b103c50eb3 P 43736346b1ff48c78ac919c80ba58654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 450, Committed index: 450, Last appended: 1.450, Last appended by leader: 450, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } }
I20240501 14:10:03.137686 13577 raft_consensus.cc:3037] T cce2260455ac4edbb9bc54b103c50eb3 P 43736346b1ff48c78ac919c80ba58654 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:03.138907 13577 raft_consensus.cc:2454] T cce2260455ac4edbb9bc54b103c50eb3 P 43736346b1ff48c78ac919c80ba58654 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e26df472efe547f8bc5951dce352471b in term 2.
I20240501 14:10:03.140061 14883 leader_election.cc:302] T cce2260455ac4edbb9bc54b103c50eb3 P e26df472efe547f8bc5951dce352471b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43736346b1ff48c78ac919c80ba58654, e26df472efe547f8bc5951dce352471b; no voters:
I20240501 14:10:03.140252 16880 raft_consensus.cc:2781] T cce2260455ac4edbb9bc54b103c50eb3 P e26df472efe547f8bc5951dce352471b [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:10:03.140414 16880 raft_consensus.cc:686] T cce2260455ac4edbb9bc54b103c50eb3 P e26df472efe547f8bc5951dce352471b [term 2 LEADER]: Becoming Leader. State: Replica: e26df472efe547f8bc5951dce352471b, State: Running, Role: LEADER
I20240501 14:10:03.140549 16880 consensus_queue.cc:237] T cce2260455ac4edbb9bc54b103c50eb3 P e26df472efe547f8bc5951dce352471b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 450, Committed index: 450, Last appended: 1.450, Last appended by leader: 450, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } }
I20240501 14:10:03.143353 13299 catalog_manager.cc:5484] T cce2260455ac4edbb9bc54b103c50eb3 P e26df472efe547f8bc5951dce352471b reported cstate change: term changed from 1 to 2, leader changed from 43736346b1ff48c78ac919c80ba58654 (127.7.39.130) to e26df472efe547f8bc5951dce352471b (127.7.39.134). New cstate: current_term: 2 leader_uuid: "e26df472efe547f8bc5951dce352471b" committed_config { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } }
I20240501 14:10:03.147279 13443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cce2260455ac4edbb9bc54b103c50eb3" candidate_uuid: "e26df472efe547f8bc5951dce352471b" candidate_term: 2 candidate_status { last_received { term: 1 index: 450 } } ignore_live_leader: true dest_uuid: "50d514f034204fc38331aa0a4fd74a1b"
I20240501 14:10:03.147403 13443 raft_consensus.cc:3037] T cce2260455ac4edbb9bc54b103c50eb3 P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:03.148460 13443 raft_consensus.cc:2454] T cce2260455ac4edbb9bc54b103c50eb3 P 50d514f034204fc38331aa0a4fd74a1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e26df472efe547f8bc5951dce352471b in term 2.
W20240501 14:10:03.155427 13534 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60886: Illegal state: replica 43736346b1ff48c78ac919c80ba58654 is not leader of this config: current role FOLLOWER
W20240501 14:10:03.202701 13405 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60894: Illegal state: replica 50d514f034204fc38331aa0a4fd74a1b is not leader of this config: current role FOLLOWER
I20240501 14:10:03.249907 13577 raft_consensus.cc:1261] T cce2260455ac4edbb9bc54b103c50eb3 P 43736346b1ff48c78ac919c80ba58654 [term 2 FOLLOWER]: Refusing update from remote peer e26df472efe547f8bc5951dce352471b: Log matching property violated. Preceding OpId in replica: term: 1 index: 450. Preceding OpId from leader: term: 2 index: 452. (index mismatch)
I20240501 14:10:03.249979 13446 raft_consensus.cc:1261] T cce2260455ac4edbb9bc54b103c50eb3 P 50d514f034204fc38331aa0a4fd74a1b [term 2 FOLLOWER]: Refusing update from remote peer e26df472efe547f8bc5951dce352471b: Log matching property violated. Preceding OpId in replica: term: 1 index: 450. Preceding OpId from leader: term: 2 index: 452. (index mismatch)
I20240501 14:10:03.250399 16880 consensus_queue.cc:1035] T cce2260455ac4edbb9bc54b103c50eb3 P e26df472efe547f8bc5951dce352471b [LEADER]: Connected to new peer: Peer: permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 451, Last known committed idx: 450, Time since last communication: 0.000s
I20240501 14:10:03.250560 16930 consensus_queue.cc:1035] T cce2260455ac4edbb9bc54b103c50eb3 P e26df472efe547f8bc5951dce352471b [LEADER]: Connected to new peer: Peer: permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 451, Last known committed idx: 450, Time since last communication: 0.000s
I20240501 14:10:03.256790 16398 mvcc.cc:205] Tried to move back new op lower bound from 7022889382907437056 to 7022889382465396736. Current Snapshot: MvccSnapshot[applied={T|T < 7022889382907437056}]
I20240501 14:10:03.262462 16527 mvcc.cc:205] Tried to move back new op lower bound from 7022889382907437056 to 7022889382465396736. Current Snapshot: MvccSnapshot[applied={T|T < 7022889382907437056 or (T in {7022889382907437056})}]
I20240501 14:10:03.386839 14665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:03.387447 14928 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:03.392104 15062 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:03.394197 13687 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:03.401093 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:03.429478 13555 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:03.455024 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:03.681135 14685 consensus_queue.cc:237] T cde568b4e46a4c5499031d0e18ae51de P 2496e4af45ba458eacaacc9d25dcf01a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 444, Committed index: 444, Last appended: 2.445, Last appended by leader: 311, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 446 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "2496e4af45ba458eacaacc9d25dcf01a" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 43977 } }
I20240501 14:10:03.690672 13446 raft_consensus.cc:1261] T cde568b4e46a4c5499031d0e18ae51de P 50d514f034204fc38331aa0a4fd74a1b [term 2 FOLLOWER]: Refusing update from remote peer 2496e4af45ba458eacaacc9d25dcf01a: Log matching property violated. Preceding OpId in replica: term: 2 index: 445. Preceding OpId from leader: term: 2 index: 446. (index mismatch)
I20240501 14:10:03.692273 16580 consensus_queue.cc:1035] T cde568b4e46a4c5499031d0e18ae51de P 2496e4af45ba458eacaacc9d25dcf01a [LEADER]: Connected to new peer: Peer: permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 446, Last known committed idx: 444, Time since last communication: 0.000s
I20240501 14:10:03.698261 16580 raft_consensus.cc:2932] T cde568b4e46a4c5499031d0e18ae51de P 2496e4af45ba458eacaacc9d25dcf01a [term 2 LEADER]: Committing config change with OpId 2.446: config changed from index 23 to 446, VOTER a278448771e14471ab4df1e6649485bd (127.7.39.131) evicted. New config: { opid_index: 446 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "2496e4af45ba458eacaacc9d25dcf01a" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 43977 } } }
I20240501 14:10:03.698640 13443 raft_consensus.cc:2932] T cde568b4e46a4c5499031d0e18ae51de P 50d514f034204fc38331aa0a4fd74a1b [term 2 FOLLOWER]: Committing config change with OpId 2.446: config changed from index 23 to 446, VOTER a278448771e14471ab4df1e6649485bd (127.7.39.131) evicted. New config: { opid_index: 446 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "2496e4af45ba458eacaacc9d25dcf01a" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 43977 } } }
I20240501 14:10:03.701159 13299 catalog_manager.cc:5484] T cde568b4e46a4c5499031d0e18ae51de P 50d514f034204fc38331aa0a4fd74a1b reported cstate change: config changed from index 23 to 446, VOTER a278448771e14471ab4df1e6649485bd (127.7.39.131) evicted. New cstate: current_term: 2 leader_uuid: "2496e4af45ba458eacaacc9d25dcf01a" committed_config { opid_index: 446 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "2496e4af45ba458eacaacc9d25dcf01a" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 43977 } } }
I20240501 14:10:03.729359 13687 tablet_service.cc:1514] Processing DeleteTablet for tablet cde568b4e46a4c5499031d0e18ae51de with delete_type TABLET_DATA_TOMBSTONED (TS a278448771e14471ab4df1e6649485bd not found in new config with opid_index 446) from {username='slave'} at 127.0.0.1:45074
I20240501 14:10:03.763646 17013 tablet_replica.cc:330] T cde568b4e46a4c5499031d0e18ae51de P a278448771e14471ab4df1e6649485bd: stopping tablet replica
I20240501 14:10:03.763833 17013 raft_consensus.cc:2229] T cde568b4e46a4c5499031d0e18ae51de P a278448771e14471ab4df1e6649485bd [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:03.763947 17013 pending_rounds.cc:62] T cde568b4e46a4c5499031d0e18ae51de P a278448771e14471ab4df1e6649485bd: Trying to abort 1 pending ops.
I20240501 14:10:03.763981 17013 pending_rounds.cc:68] T cde568b4e46a4c5499031d0e18ae51de P a278448771e14471ab4df1e6649485bd: Aborting op as it isn't in flight: id { term: 2 index: 445 } timestamp: 7022889384662224896 op_type: WRITE_OP write_request { tablet_id: "cde568b4e46a4c5499031d0e18ae51de" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\222\342U!\264\346\235\341\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""299d0a9d8a47ff2d" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7022889384618745856 authz_token { token_data: "\010\326\234\311\261\006\"3\n\005slave\022*\n 7319ca2094424c85b7008d8839665ed4\020\001\030\001 \001(\001" signature: "<redacted>""\025\325\266\004\370Q>\220\3055Mm\263\366b6\361\210W\374\267a[\r.< -\325\304\375\207\255\227Z\363\225(,s\014\346P\021Qd\365\010\\c\272\365 \362\374\375W\022?\247\314\263\237\373" signing_key_seq_num: 0 } } request_id { client_id: "3c09af551a4b430e8186a8660a089342" seq_no: 1262 first_incomplete_seq_no: 1262 attempt_no: 0 }
I20240501 14:10:03.764227 17013 raft_consensus.cc:2258] T cde568b4e46a4c5499031d0e18ae51de P a278448771e14471ab4df1e6649485bd [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:03.774466 17013 ts_tablet_manager.cc:1875] T cde568b4e46a4c5499031d0e18ae51de P a278448771e14471ab4df1e6649485bd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:03.787663 17013 ts_tablet_manager.cc:1888] T cde568b4e46a4c5499031d0e18ae51de P a278448771e14471ab4df1e6649485bd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.445
I20240501 14:10:03.787752 17013 log.cc:1196] T cde568b4e46a4c5499031d0e18ae51de P a278448771e14471ab4df1e6649485bd: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal/wals/cde568b4e46a4c5499031d0e18ae51de
I20240501 14:10:03.789026 13288 catalog_manager.cc:4830] TS a278448771e14471ab4df1e6649485bd (127.7.39.131:46461): tablet cde568b4e46a4c5499031d0e18ae51de (table rebalance_test_table_4 [id=7319ca2094424c85b7008d8839665ed4]) successfully deleted
I20240501 14:10:03.993875 13577 consensus_queue.cc:237] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 403, Committed index: 403, Last appended: 1.403, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 404 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } }
W20240501 14:10:03.995507 13511 consensus_peers.cc:485] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654 -> Peer 39bde7c058154f8aa40085db77f8c6e5 (127.7.39.135:37645): Couldn't send request to peer 39bde7c058154f8aa40085db77f8c6e5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f1c30f16a13444cc95132eeb9ea6c96c. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:03.995669 13443 raft_consensus.cc:1261] T f1c30f16a13444cc95132eeb9ea6c96c P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Refusing update from remote peer 43736346b1ff48c78ac919c80ba58654: Log matching property violated. Preceding OpId in replica: term: 1 index: 403. Preceding OpId from leader: term: 1 index: 404. (index mismatch)
I20240501 14:10:03.996027 16769 consensus_queue.cc:1035] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 404, Last known committed idx: 403, Time since last communication: 0.000s
I20240501 14:10:03.998443 16985 raft_consensus.cc:2932] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654 [term 1 LEADER]: Committing config change with OpId 1.404: config changed from index -1 to 404, VOTER 39bde7c058154f8aa40085db77f8c6e5 (127.7.39.135) added. New config: { opid_index: 404 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } } }
I20240501 14:10:04.000396 13443 raft_consensus.cc:2932] T f1c30f16a13444cc95132eeb9ea6c96c P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Committing config change with OpId 1.404: config changed from index -1 to 404, VOTER 39bde7c058154f8aa40085db77f8c6e5 (127.7.39.135) added. New config: { opid_index: 404 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } } }
I20240501 14:10:04.003949 13301 catalog_manager.cc:5484] T f1c30f16a13444cc95132eeb9ea6c96c P 50d514f034204fc38331aa0a4fd74a1b reported cstate change: config changed from index -1 to 404, VOTER 39bde7c058154f8aa40085db77f8c6e5 (127.7.39.135) added. New cstate: current_term: 1 leader_uuid: "43736346b1ff48c78ac919c80ba58654" committed_config { opid_index: 404 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } } }
I20240501 14:10:04.293478 14665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:04.311913 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:04.319198 13687 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:04.327184 13555 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:04.327195 15062 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:04.332161 14928 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:04.332442 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:04.465533 17089 ts_tablet_manager.cc:902] T f1c30f16a13444cc95132eeb9ea6c96c P 39bde7c058154f8aa40085db77f8c6e5: Initiating tablet copy from peer 43736346b1ff48c78ac919c80ba58654 (127.7.39.130:43311)
I20240501 14:10:04.469098 17089 tablet_copy_client.cc:308] T f1c30f16a13444cc95132eeb9ea6c96c P 39bde7c058154f8aa40085db77f8c6e5: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.130:43311
I20240501 14:10:04.470788 13597 tablet_copy_service.cc:140] P 43736346b1ff48c78ac919c80ba58654: Received BeginTabletCopySession request for tablet f1c30f16a13444cc95132eeb9ea6c96c from peer 39bde7c058154f8aa40085db77f8c6e5 ({username='slave'} at 127.7.39.135:52733)
I20240501 14:10:04.470883 13597 tablet_copy_service.cc:161] P 43736346b1ff48c78ac919c80ba58654: Beginning new tablet copy session on tablet f1c30f16a13444cc95132eeb9ea6c96c from peer 39bde7c058154f8aa40085db77f8c6e5 at {username='slave'} at 127.7.39.135:52733: session id = 39bde7c058154f8aa40085db77f8c6e5-f1c30f16a13444cc95132eeb9ea6c96c
I20240501 14:10:04.471781 13597 tablet_copy_source_session.cc:205] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:04.472644 17089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1c30f16a13444cc95132eeb9ea6c96c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:04.476410 17089 tablet_copy_client.cc:783] T f1c30f16a13444cc95132eeb9ea6c96c P 39bde7c058154f8aa40085db77f8c6e5: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:04.476954 17089 tablet_copy_client.cc:647] T f1c30f16a13444cc95132eeb9ea6c96c P 39bde7c058154f8aa40085db77f8c6e5: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:04.485392 17089 tablet_copy_client.cc:523] T f1c30f16a13444cc95132eeb9ea6c96c P 39bde7c058154f8aa40085db77f8c6e5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:04.495354 17089 tablet_bootstrap.cc:492] T f1c30f16a13444cc95132eeb9ea6c96c P 39bde7c058154f8aa40085db77f8c6e5: Bootstrap starting.
I20240501 14:10:04.701252 15062 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:04.724115 13555 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:04.724714 14928 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:04.726155 13687 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:04.743158 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:04.758275 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:04.806604 14665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:04.988392 17089 tablet_bootstrap.cc:492] T f1c30f16a13444cc95132eeb9ea6c96c P 39bde7c058154f8aa40085db77f8c6e5: Bootstrap replayed 1/1 log segments. Stats: ops{read=412 overwritten=0 applied=411 ignored=0} inserts{seen=1083 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 14:10:04.988762 17089 tablet_bootstrap.cc:492] T f1c30f16a13444cc95132eeb9ea6c96c P 39bde7c058154f8aa40085db77f8c6e5: Bootstrap complete.
I20240501 14:10:04.989547 17089 ts_tablet_manager.cc:1367] T f1c30f16a13444cc95132eeb9ea6c96c P 39bde7c058154f8aa40085db77f8c6e5: Time spent bootstrapping tablet: real 0.494s user 0.299s sys 0.012s
I20240501 14:10:04.990144 17089 raft_consensus.cc:348] T f1c30f16a13444cc95132eeb9ea6c96c P 39bde7c058154f8aa40085db77f8c6e5 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 404 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } }
I20240501 14:10:04.990453 17089 raft_consensus.cc:729] T f1c30f16a13444cc95132eeb9ea6c96c P 39bde7c058154f8aa40085db77f8c6e5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39bde7c058154f8aa40085db77f8c6e5, State: Initialized, Role: FOLLOWER
I20240501 14:10:04.990640 17089 consensus_queue.cc:260] T f1c30f16a13444cc95132eeb9ea6c96c P 39bde7c058154f8aa40085db77f8c6e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 411, Last appended: 1.412, Last appended by leader: 412, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 404 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } }
I20240501 14:10:04.990939 17089 ts_tablet_manager.cc:1398] T f1c30f16a13444cc95132eeb9ea6c96c P 39bde7c058154f8aa40085db77f8c6e5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:04.992017 13597 tablet_copy_service.cc:342] P 43736346b1ff48c78ac919c80ba58654: Request end of tablet copy session 39bde7c058154f8aa40085db77f8c6e5-f1c30f16a13444cc95132eeb9ea6c96c received from {username='slave'} at 127.7.39.135:52733
I20240501 14:10:04.992105 13597 tablet_copy_service.cc:433] P 43736346b1ff48c78ac919c80ba58654: ending tablet copy session 39bde7c058154f8aa40085db77f8c6e5-f1c30f16a13444cc95132eeb9ea6c96c on tablet f1c30f16a13444cc95132eeb9ea6c96c with peer 39bde7c058154f8aa40085db77f8c6e5
I20240501 14:10:05.056308 14948 consensus_queue.cc:237] T cce2260455ac4edbb9bc54b103c50eb3 P e26df472efe547f8bc5951dce352471b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 526, Committed index: 526, Last appended: 2.526, Last appended by leader: 450, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 527 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } }
I20240501 14:10:05.057394 13443 raft_consensus.cc:1261] T cce2260455ac4edbb9bc54b103c50eb3 P 50d514f034204fc38331aa0a4fd74a1b [term 2 FOLLOWER]: Refusing update from remote peer e26df472efe547f8bc5951dce352471b: Log matching property violated. Preceding OpId in replica: term: 2 index: 526. Preceding OpId from leader: term: 2 index: 527. (index mismatch)
I20240501 14:10:05.057930 16962 consensus_queue.cc:1035] T cce2260455ac4edbb9bc54b103c50eb3 P e26df472efe547f8bc5951dce352471b [LEADER]: Connected to new peer: Peer: permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 527, Last known committed idx: 526, Time since last communication: 0.000s
I20240501 14:10:05.066942 16962 raft_consensus.cc:2932] T cce2260455ac4edbb9bc54b103c50eb3 P e26df472efe547f8bc5951dce352471b [term 2 LEADER]: Committing config change with OpId 2.527: config changed from index 74 to 527, VOTER 43736346b1ff48c78ac919c80ba58654 (127.7.39.130) evicted. New config: { opid_index: 527 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } }
I20240501 14:10:05.067266 13443 raft_consensus.cc:2932] T cce2260455ac4edbb9bc54b103c50eb3 P 50d514f034204fc38331aa0a4fd74a1b [term 2 FOLLOWER]: Committing config change with OpId 2.527: config changed from index 74 to 527, VOTER 43736346b1ff48c78ac919c80ba58654 (127.7.39.130) evicted. New config: { opid_index: 527 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } }
I20240501 14:10:05.069902 13301 catalog_manager.cc:5484] T cce2260455ac4edbb9bc54b103c50eb3 P e26df472efe547f8bc5951dce352471b reported cstate change: config changed from index 74 to 527, VOTER 43736346b1ff48c78ac919c80ba58654 (127.7.39.130) evicted. New cstate: current_term: 2 leader_uuid: "e26df472efe547f8bc5951dce352471b" committed_config { opid_index: 527 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } }
I20240501 14:10:05.078387 13555 tablet_service.cc:1514] Processing DeleteTablet for tablet cce2260455ac4edbb9bc54b103c50eb3 with delete_type TABLET_DATA_TOMBSTONED (TS 43736346b1ff48c78ac919c80ba58654 not found in new config with opid_index 527) from {username='slave'} at 127.0.0.1:57610
I20240501 14:10:05.090591 17159 tablet_replica.cc:330] T cce2260455ac4edbb9bc54b103c50eb3 P 43736346b1ff48c78ac919c80ba58654: stopping tablet replica
I20240501 14:10:05.090772 17159 raft_consensus.cc:2229] T cce2260455ac4edbb9bc54b103c50eb3 P 43736346b1ff48c78ac919c80ba58654 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:05.090920 17159 raft_consensus.cc:2258] T cce2260455ac4edbb9bc54b103c50eb3 P 43736346b1ff48c78ac919c80ba58654 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:05.092401 17159 ts_tablet_manager.cc:1875] T cce2260455ac4edbb9bc54b103c50eb3 P 43736346b1ff48c78ac919c80ba58654: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:05.094933 17159 ts_tablet_manager.cc:1888] T cce2260455ac4edbb9bc54b103c50eb3 P 43736346b1ff48c78ac919c80ba58654: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.526
I20240501 14:10:05.095917 17159 log.cc:1196] T cce2260455ac4edbb9bc54b103c50eb3 P 43736346b1ff48c78ac919c80ba58654: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal/wals/cce2260455ac4edbb9bc54b103c50eb3
I20240501 14:10:05.097195 13291 catalog_manager.cc:4830] TS 43736346b1ff48c78ac919c80ba58654 (127.7.39.130:43311): tablet cce2260455ac4edbb9bc54b103c50eb3 (table rebalance_test_table_3 [id=50bc5b382cee4078bfb9cc86c185c1b2]) successfully deleted
I20240501 14:10:05.342708 15062 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:05.343832 13555 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:05.344441 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:05.347465 14665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:05.349283 13687 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:05.350615 14928 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:05.377007 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:05.554949 15081 raft_consensus.cc:1203] T f1c30f16a13444cc95132eeb9ea6c96c P 39bde7c058154f8aa40085db77f8c6e5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.403->[1.404-1.444] Dedup: 1.412->[1.413-1.444]
I20240501 14:10:05.794879 15062 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:05.803689 14665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:05.826009 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:05.843756 13687 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:05.843807 14928 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:05.844071 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:05.896485 13555 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:06.072836 13577 consensus_queue.cc:237] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 461, Committed index: 461, Last appended: 1.461, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 462 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } }
I20240501 14:10:06.074440 15082 raft_consensus.cc:1261] T f1c30f16a13444cc95132eeb9ea6c96c P 39bde7c058154f8aa40085db77f8c6e5 [term 1 FOLLOWER]: Refusing update from remote peer 43736346b1ff48c78ac919c80ba58654: Log matching property violated. Preceding OpId in replica: term: 1 index: 461. Preceding OpId from leader: term: 1 index: 462. (index mismatch)
I20240501 14:10:06.077411 16769 consensus_queue.cc:1035] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 462, Last known committed idx: 461, Time since last communication: 0.000s
I20240501 14:10:06.080214 15180 raft_consensus.cc:2932] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654 [term 1 LEADER]: Committing config change with OpId 1.462: config changed from index 404 to 462, VOTER 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129) evicted. New config: { opid_index: 462 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } } }
I20240501 14:10:06.083654 15082 raft_consensus.cc:2932] T f1c30f16a13444cc95132eeb9ea6c96c P 39bde7c058154f8aa40085db77f8c6e5 [term 1 FOLLOWER]: Committing config change with OpId 1.462: config changed from index 404 to 462, VOTER 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129) evicted. New config: { opid_index: 462 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } } }
I20240501 14:10:06.086612 13301 catalog_manager.cc:5484] T f1c30f16a13444cc95132eeb9ea6c96c P 39bde7c058154f8aa40085db77f8c6e5 reported cstate change: config changed from index 404 to 462, VOTER 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129) evicted. New cstate: current_term: 1 leader_uuid: "43736346b1ff48c78ac919c80ba58654" committed_config { opid_index: 462 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } } }
I20240501 14:10:06.101219 13426 tablet_service.cc:1514] Processing DeleteTablet for tablet f1c30f16a13444cc95132eeb9ea6c96c with delete_type TABLET_DATA_TOMBSTONED (TS 50d514f034204fc38331aa0a4fd74a1b not found in new config with opid_index 462) from {username='slave'} at 127.0.0.1:45262
I20240501 14:10:06.102416 17283 tablet_replica.cc:330] T f1c30f16a13444cc95132eeb9ea6c96c P 50d514f034204fc38331aa0a4fd74a1b: stopping tablet replica
I20240501 14:10:06.102871 17283 raft_consensus.cc:2229] T f1c30f16a13444cc95132eeb9ea6c96c P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:06.103066 17283 raft_consensus.cc:2258] T f1c30f16a13444cc95132eeb9ea6c96c P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:06.103801 17283 ts_tablet_manager.cc:1875] T f1c30f16a13444cc95132eeb9ea6c96c P 50d514f034204fc38331aa0a4fd74a1b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:06.106290 17283 ts_tablet_manager.cc:1888] T f1c30f16a13444cc95132eeb9ea6c96c P 50d514f034204fc38331aa0a4fd74a1b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.461
I20240501 14:10:06.106436 17283 log.cc:1196] T f1c30f16a13444cc95132eeb9ea6c96c P 50d514f034204fc38331aa0a4fd74a1b: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal/wals/f1c30f16a13444cc95132eeb9ea6c96c
I20240501 14:10:06.109159 13291 catalog_manager.cc:4830] TS 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129:37933): tablet f1c30f16a13444cc95132eeb9ea6c96c (table rebalance_test_table_4 [id=7319ca2094424c85b7008d8839665ed4]) successfully deleted
I20240501 14:10:06.527524 14665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:06.562299 15062 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:06.576409 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:06.590080 14928 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:06.602916 13687 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:06.623342 13555 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:06.697124 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:06.867089 13577 consensus_queue.cc:237] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 490, Committed index: 490, Last appended: 1.491, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 492 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } } peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } }
W20240501 14:10:06.872928 13511 consensus_peers.cc:485] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654 -> Peer e26df472efe547f8bc5951dce352471b (127.7.39.134:33035): Couldn't send request to peer e26df472efe547f8bc5951dce352471b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f1c30f16a13444cc95132eeb9ea6c96c. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:06.873332 15080 raft_consensus.cc:1261] T f1c30f16a13444cc95132eeb9ea6c96c P 39bde7c058154f8aa40085db77f8c6e5 [term 1 FOLLOWER]: Refusing update from remote peer 43736346b1ff48c78ac919c80ba58654: Log matching property violated. Preceding OpId in replica: term: 1 index: 490. Preceding OpId from leader: term: 1 index: 492. (index mismatch)
I20240501 14:10:06.874737 16986 consensus_queue.cc:1035] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 492, Last known committed idx: 490, Time since last communication: 0.000s
I20240501 14:10:06.876446 15079 raft_consensus.cc:1203] T f1c30f16a13444cc95132eeb9ea6c96c P 39bde7c058154f8aa40085db77f8c6e5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.490->[1.491-1.492] Dedup: 1.491->[1.492-1.492]
I20240501 14:10:06.879330 17296 raft_consensus.cc:2932] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654 [term 1 LEADER]: Committing config change with OpId 1.492: config changed from index 462 to 492, VOTER e26df472efe547f8bc5951dce352471b (127.7.39.134) added. New config: { opid_index: 492 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } } peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } }
I20240501 14:10:06.881386 15079 raft_consensus.cc:2932] T f1c30f16a13444cc95132eeb9ea6c96c P 39bde7c058154f8aa40085db77f8c6e5 [term 1 FOLLOWER]: Committing config change with OpId 1.492: config changed from index 462 to 492, VOTER e26df472efe547f8bc5951dce352471b (127.7.39.134) added. New config: { opid_index: 492 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } } peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } }
I20240501 14:10:06.884274 13301 catalog_manager.cc:5484] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654 reported cstate change: config changed from index 462 to 492, VOTER e26df472efe547f8bc5951dce352471b (127.7.39.134) added. New cstate: current_term: 1 leader_uuid: "43736346b1ff48c78ac919c80ba58654" committed_config { opid_index: 492 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } } peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } }
I20240501 14:10:07.085415 13555 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:07.096573 13687 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:07.103017 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:07.114285 15062 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:07.119464 14928 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:07.124054 14665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:07.172761 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:07.437513 17422 ts_tablet_manager.cc:902] T f1c30f16a13444cc95132eeb9ea6c96c P e26df472efe547f8bc5951dce352471b: Initiating tablet copy from peer 43736346b1ff48c78ac919c80ba58654 (127.7.39.130:43311)
I20240501 14:10:07.438092 17422 tablet_copy_client.cc:308] T f1c30f16a13444cc95132eeb9ea6c96c P e26df472efe547f8bc5951dce352471b: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.130:43311
I20240501 14:10:07.439142 13597 tablet_copy_service.cc:140] P 43736346b1ff48c78ac919c80ba58654: Received BeginTabletCopySession request for tablet f1c30f16a13444cc95132eeb9ea6c96c from peer e26df472efe547f8bc5951dce352471b ({username='slave'} at 127.7.39.134:45565)
I20240501 14:10:07.439237 13597 tablet_copy_service.cc:161] P 43736346b1ff48c78ac919c80ba58654: Beginning new tablet copy session on tablet f1c30f16a13444cc95132eeb9ea6c96c from peer e26df472efe547f8bc5951dce352471b at {username='slave'} at 127.7.39.134:45565: session id = e26df472efe547f8bc5951dce352471b-f1c30f16a13444cc95132eeb9ea6c96c
I20240501 14:10:07.440183 13597 tablet_copy_source_session.cc:205] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:07.445152 17422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1c30f16a13444cc95132eeb9ea6c96c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:07.455385 17422 tablet_copy_client.cc:783] T f1c30f16a13444cc95132eeb9ea6c96c P e26df472efe547f8bc5951dce352471b: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:07.456189 17422 tablet_copy_client.cc:647] T f1c30f16a13444cc95132eeb9ea6c96c P e26df472efe547f8bc5951dce352471b: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:07.468305 17422 tablet_copy_client.cc:523] T f1c30f16a13444cc95132eeb9ea6c96c P e26df472efe547f8bc5951dce352471b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:07.469802 17422 tablet_bootstrap.cc:492] T f1c30f16a13444cc95132eeb9ea6c96c P e26df472efe547f8bc5951dce352471b: Bootstrap starting.
I20240501 14:10:07.555163 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:07.556110 15062 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:07.556777 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:07.566457 14928 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:07.573570 13687 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:07.591897 14665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:07.644301 13555 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:08.067669 17422 tablet_bootstrap.cc:492] T f1c30f16a13444cc95132eeb9ea6c96c P e26df472efe547f8bc5951dce352471b: Bootstrap replayed 1/1 log segments. Stats: ops{read=514 overwritten=0 applied=513 ignored=0} inserts{seen=1183 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 14:10:08.068154 17422 tablet_bootstrap.cc:492] T f1c30f16a13444cc95132eeb9ea6c96c P e26df472efe547f8bc5951dce352471b: Bootstrap complete.
I20240501 14:10:08.068492 17422 ts_tablet_manager.cc:1367] T f1c30f16a13444cc95132eeb9ea6c96c P e26df472efe547f8bc5951dce352471b: Time spent bootstrapping tablet: real 0.599s user 0.381s sys 0.012s
I20240501 14:10:08.069037 17422 raft_consensus.cc:348] T f1c30f16a13444cc95132eeb9ea6c96c P e26df472efe547f8bc5951dce352471b [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 492 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } } peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } }
I20240501 14:10:08.069737 17422 raft_consensus.cc:729] T f1c30f16a13444cc95132eeb9ea6c96c P e26df472efe547f8bc5951dce352471b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e26df472efe547f8bc5951dce352471b, State: Initialized, Role: FOLLOWER
I20240501 14:10:08.069979 17422 consensus_queue.cc:260] T f1c30f16a13444cc95132eeb9ea6c96c P e26df472efe547f8bc5951dce352471b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 513, Last appended: 1.514, Last appended by leader: 514, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 492 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } } peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } }
I20240501 14:10:08.070353 17422 ts_tablet_manager.cc:1398] T f1c30f16a13444cc95132eeb9ea6c96c P e26df472efe547f8bc5951dce352471b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:10:08.070869 13597 tablet_copy_service.cc:342] P 43736346b1ff48c78ac919c80ba58654: Request end of tablet copy session e26df472efe547f8bc5951dce352471b-f1c30f16a13444cc95132eeb9ea6c96c received from {username='slave'} at 127.7.39.134:45565
I20240501 14:10:08.070952 13597 tablet_copy_service.cc:433] P 43736346b1ff48c78ac919c80ba58654: ending tablet copy session e26df472efe547f8bc5951dce352471b-f1c30f16a13444cc95132eeb9ea6c96c on tablet f1c30f16a13444cc95132eeb9ea6c96c with peer e26df472efe547f8bc5951dce352471b
I20240501 14:10:08.333119 14665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:08.336190 15062 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:08.346527 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:08.364442 13555 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:08.400418 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:08.426681 13687 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:08.464488 14928 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:08.509831 14946 raft_consensus.cc:1203] T f1c30f16a13444cc95132eeb9ea6c96c P e26df472efe547f8bc5951dce352471b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.491->[1.492-1.556] Dedup: 1.514->[1.515-1.556]
I20240501 14:10:08.682511 14665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:08.750537 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:08.764518 15062 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:08.765671 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:08.766165 13687 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:08.775903 13555 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:08.806488 14928 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:09.361053 14665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:09.414319 13687 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:09.417433 15062 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:09.439805 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:09.442817 14928 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:09.447535 13555 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:09.457253 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:09.576556 13577 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f1c30f16a13444cc95132eeb9ea6c96c"
dest_uuid: "43736346b1ff48c78ac919c80ba58654"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47314
I20240501 14:10:09.576714 13577 raft_consensus.cc:595] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:10:09.589286 16985 raft_consensus.cc:979] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654: : Instructing follower e26df472efe547f8bc5951dce352471b to start an election
I20240501 14:10:09.589401 16985 raft_consensus.cc:1067] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654 [term 1 LEADER]: Signalling peer e26df472efe547f8bc5951dce352471b to start an election
I20240501 14:10:09.593187 14946 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f1c30f16a13444cc95132eeb9ea6c96c"
dest_uuid: "e26df472efe547f8bc5951dce352471b"
from {username='slave'} at 127.7.39.130:39903
I20240501 14:10:09.593319 14946 raft_consensus.cc:482] T f1c30f16a13444cc95132eeb9ea6c96c P e26df472efe547f8bc5951dce352471b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:10:09.593379 14946 raft_consensus.cc:3037] T f1c30f16a13444cc95132eeb9ea6c96c P e26df472efe547f8bc5951dce352471b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:09.594475 14946 raft_consensus.cc:504] T f1c30f16a13444cc95132eeb9ea6c96c P e26df472efe547f8bc5951dce352471b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 492 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } } peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } }
I20240501 14:10:09.595641 14946 leader_election.cc:288] T f1c30f16a13444cc95132eeb9ea6c96c P e26df472efe547f8bc5951dce352471b [CANDIDATE]: Term 2 election: Requested vote from peers 43736346b1ff48c78ac919c80ba58654 (127.7.39.130:43311), 39bde7c058154f8aa40085db77f8c6e5 (127.7.39.135:37645)
I20240501 14:10:09.595764 13577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1c30f16a13444cc95132eeb9ea6c96c" candidate_uuid: "e26df472efe547f8bc5951dce352471b" candidate_term: 2 candidate_status { last_received { term: 1 index: 588 } } ignore_live_leader: true dest_uuid: "43736346b1ff48c78ac919c80ba58654"
I20240501 14:10:09.595893 13577 raft_consensus.cc:3032] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:10:09.595929 13577 raft_consensus.cc:729] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 43736346b1ff48c78ac919c80ba58654, State: Running, Role: LEADER
I20240501 14:10:09.596050 13577 consensus_queue.cc:260] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 588, Committed index: 588, Last appended: 1.588, Last appended by leader: 588, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 492 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } } peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } }
I20240501 14:10:09.596139 13577 raft_consensus.cc:3037] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:09.596355 14946 raft_consensus.cc:1226] T f1c30f16a13444cc95132eeb9ea6c96c P e26df472efe547f8bc5951dce352471b [term 2 FOLLOWER]: Rejecting Update request from peer 43736346b1ff48c78ac919c80ba58654 for earlier term 1. Current term is 2. Ops: []
I20240501 14:10:09.597136 13577 raft_consensus.cc:2454] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e26df472efe547f8bc5951dce352471b in term 2.
I20240501 14:10:09.597456 14883 leader_election.cc:302] T f1c30f16a13444cc95132eeb9ea6c96c P e26df472efe547f8bc5951dce352471b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43736346b1ff48c78ac919c80ba58654, e26df472efe547f8bc5951dce352471b; no voters:
I20240501 14:10:09.597635 16930 raft_consensus.cc:2781] T f1c30f16a13444cc95132eeb9ea6c96c P e26df472efe547f8bc5951dce352471b [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:10:09.597704 16930 raft_consensus.cc:686] T f1c30f16a13444cc95132eeb9ea6c96c P e26df472efe547f8bc5951dce352471b [term 2 LEADER]: Becoming Leader. State: Replica: e26df472efe547f8bc5951dce352471b, State: Running, Role: LEADER
I20240501 14:10:09.597855 16930 consensus_queue.cc:237] T f1c30f16a13444cc95132eeb9ea6c96c P e26df472efe547f8bc5951dce352471b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 587, Committed index: 587, Last appended: 1.588, Last appended by leader: 588, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 492 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } } peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } }
I20240501 14:10:09.599555 13295 catalog_manager.cc:5484] T f1c30f16a13444cc95132eeb9ea6c96c P e26df472efe547f8bc5951dce352471b reported cstate change: term changed from 1 to 2, leader changed from 43736346b1ff48c78ac919c80ba58654 (127.7.39.130) to e26df472efe547f8bc5951dce352471b (127.7.39.134). New cstate: current_term: 2 leader_uuid: "e26df472efe547f8bc5951dce352471b" committed_config { opid_index: 492 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } } peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } }
I20240501 14:10:09.618849 15079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1c30f16a13444cc95132eeb9ea6c96c" candidate_uuid: "e26df472efe547f8bc5951dce352471b" candidate_term: 2 candidate_status { last_received { term: 1 index: 588 } } ignore_live_leader: true dest_uuid: "39bde7c058154f8aa40085db77f8c6e5"
I20240501 14:10:09.619077 15079 raft_consensus.cc:3037] T f1c30f16a13444cc95132eeb9ea6c96c P 39bde7c058154f8aa40085db77f8c6e5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:09.620285 15079 raft_consensus.cc:2454] T f1c30f16a13444cc95132eeb9ea6c96c P 39bde7c058154f8aa40085db77f8c6e5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e26df472efe547f8bc5951dce352471b in term 2.
W20240501 14:10:09.635857 13536 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60898: Illegal state: replica 43736346b1ff48c78ac919c80ba58654 is not leader of this config: current role FOLLOWER
I20240501 14:10:09.672351 15081 raft_consensus.cc:1261] T f1c30f16a13444cc95132eeb9ea6c96c P 39bde7c058154f8aa40085db77f8c6e5 [term 2 FOLLOWER]: Refusing update from remote peer e26df472efe547f8bc5951dce352471b: Log matching property violated. Preceding OpId in replica: term: 1 index: 588. Preceding OpId from leader: term: 2 index: 590. (index mismatch)
I20240501 14:10:09.672788 16930 consensus_queue.cc:1035] T f1c30f16a13444cc95132eeb9ea6c96c P e26df472efe547f8bc5951dce352471b [LEADER]: Connected to new peer: Peer: permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 589, Last known committed idx: 587, Time since last communication: 0.000s
I20240501 14:10:09.673229 13577 raft_consensus.cc:1261] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654 [term 2 FOLLOWER]: Refusing update from remote peer e26df472efe547f8bc5951dce352471b: Log matching property violated. Preceding OpId in replica: term: 1 index: 588. Preceding OpId from leader: term: 2 index: 590. (index mismatch)
I20240501 14:10:09.678493 16962 consensus_queue.cc:1035] T f1c30f16a13444cc95132eeb9ea6c96c P e26df472efe547f8bc5951dce352471b [LEADER]: Connected to new peer: Peer: permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 589, Last known committed idx: 588, Time since last communication: 0.000s
I20240501 14:10:09.680584 17441 mvcc.cc:205] Tried to move back new op lower bound from 7022889409212268544 to 7022889408914292736. Current Snapshot: MvccSnapshot[applied={T|T < 7022889409212268544 or (T in {7022889409212268544})}]
I20240501 14:10:09.754493 15062 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:09.754791 14665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:09.763504 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:09.814500 13687 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:09.832684 14928 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:09.842693 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:09.858217 13555 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:10.036221 14948 consensus_queue.cc:237] T f1c30f16a13444cc95132eeb9ea6c96c P e26df472efe547f8bc5951dce352471b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 602, Committed index: 602, Last appended: 2.603, Last appended by leader: 588, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 604 OBSOLETE_local: false peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } } peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } }
I20240501 14:10:10.037696 15080 raft_consensus.cc:1261] T f1c30f16a13444cc95132eeb9ea6c96c P 39bde7c058154f8aa40085db77f8c6e5 [term 2 FOLLOWER]: Refusing update from remote peer e26df472efe547f8bc5951dce352471b: Log matching property violated. Preceding OpId in replica: term: 2 index: 603. Preceding OpId from leader: term: 2 index: 604. (index mismatch)
I20240501 14:10:10.038183 17637 consensus_queue.cc:1035] T f1c30f16a13444cc95132eeb9ea6c96c P e26df472efe547f8bc5951dce352471b [LEADER]: Connected to new peer: Peer: permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 604, Last known committed idx: 602, Time since last communication: 0.000s
I20240501 14:10:10.042146 17637 raft_consensus.cc:2932] T f1c30f16a13444cc95132eeb9ea6c96c P e26df472efe547f8bc5951dce352471b [term 2 LEADER]: Committing config change with OpId 2.604: config changed from index 492 to 604, VOTER 43736346b1ff48c78ac919c80ba58654 (127.7.39.130) evicted. New config: { opid_index: 604 OBSOLETE_local: false peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } } peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } }
I20240501 14:10:10.045217 15079 raft_consensus.cc:2932] T f1c30f16a13444cc95132eeb9ea6c96c P 39bde7c058154f8aa40085db77f8c6e5 [term 2 FOLLOWER]: Committing config change with OpId 2.604: config changed from index 492 to 604, VOTER 43736346b1ff48c78ac919c80ba58654 (127.7.39.130) evicted. New config: { opid_index: 604 OBSOLETE_local: false peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } } peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } }
I20240501 14:10:10.045396 13302 catalog_manager.cc:5484] T f1c30f16a13444cc95132eeb9ea6c96c P e26df472efe547f8bc5951dce352471b reported cstate change: config changed from index 492 to 604, VOTER 43736346b1ff48c78ac919c80ba58654 (127.7.39.130) evicted. New cstate: current_term: 2 leader_uuid: "e26df472efe547f8bc5951dce352471b" committed_config { opid_index: 604 OBSOLETE_local: false peers { permanent_uuid: "39bde7c058154f8aa40085db77f8c6e5" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 37645 } } peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } }
I20240501 14:10:10.050251 13555 tablet_service.cc:1514] Processing DeleteTablet for tablet f1c30f16a13444cc95132eeb9ea6c96c with delete_type TABLET_DATA_TOMBSTONED (TS 43736346b1ff48c78ac919c80ba58654 not found in new config with opid_index 604) from {username='slave'} at 127.0.0.1:57610
I20240501 14:10:10.050951 17660 tablet_replica.cc:330] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654: stopping tablet replica
I20240501 14:10:10.051443 17660 raft_consensus.cc:2229] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:10.051581 17660 pending_rounds.cc:62] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654: Trying to abort 1 pending ops.
I20240501 14:10:10.052301 17660 pending_rounds.cc:68] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654: Aborting op as it isn't in flight: id { term: 2 index: 603 } timestamp: 7022889410697539584 op_type: WRITE_OP write_request { tablet_id: "f1c30f16a13444cc95132eeb9ea6c96c" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\264\211\200#\351\346W\354\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""070bbe882813acaa" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7022889410675339264 authz_token { token_data: "\010\326\234\311\261\006\"3\n\005slave\022*\n 7319ca2094424c85b7008d8839665ed4\020\001\030\001 \001(\001" signature: "<redacted>""\025\325\266\004\370Q>\220\3055Mm\263\366b6\361\210W\374\267a[\r.< -\325\304\375\207\255\227Z\363\225(,s\014\346P\021Qd\365\010\\c\272\365 \362\374\375W\022?\247\314\263\237\373" signing_key_seq_num: 0 } } request_id { client_id: "3c09af551a4b430e8186a8660a089342" seq_no: 1875 first_incomplete_seq_no: 1875 attempt_no: 0 }
I20240501 14:10:10.052682 17660 raft_consensus.cc:2258] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:10.057026 17660 ts_tablet_manager.cc:1875] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:10.059779 17660 ts_tablet_manager.cc:1888] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.603
I20240501 14:10:10.059911 17660 log.cc:1196] T f1c30f16a13444cc95132eeb9ea6c96c P 43736346b1ff48c78ac919c80ba58654: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal/wals/f1c30f16a13444cc95132eeb9ea6c96c
I20240501 14:10:10.060616 13291 catalog_manager.cc:4830] TS 43736346b1ff48c78ac919c80ba58654 (127.7.39.130:43311): tablet f1c30f16a13444cc95132eeb9ea6c96c (table rebalance_test_table_4 [id=7319ca2094424c85b7008d8839665ed4]) successfully deleted
I20240501 14:10:10.590919 14665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:10.609208 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:10.655884 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:10.659662 14928 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:10.670111 15062 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:10.677551 13687 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:10.755375 13555 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:11.024889 13687 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:11.025867 14665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:11.043579 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:11.069505 14928 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:11.073035 13555 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:11.084275 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:11.087121 15062 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:11.756682 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:11.757727 15062 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:11.758204 13687 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:11.763891 13555 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:11.779970 14665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:11.793689 14928 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:11.796252 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:12.168701 14665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:12.176430 15062 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:12.197118 14928 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:12.202589 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:12.203119 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:12.261216 13555 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:12.288689 13687 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:12.455677 15062 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:12.456276 14665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:12.456665 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:12.464242 14928 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:12.466792 13555 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:12.466962 13687 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:12.467231 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
f86e915e5bb5479aa7f3e095cf9884ba | 127.7.39.190:41789 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.39.148:33673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
2496e4af45ba458eacaacc9d25dcf01a | 127.7.39.132:43977 | HEALTHY | <none> | 1 | 0
39bde7c058154f8aa40085db77f8c6e5 | 127.7.39.135:37645 | HEALTHY | <none> | 1 | 0
4015e9df09424332bc3d7a9bad781641 | 127.7.39.133:42905 | HEALTHY | <none> | 0 | 0
43736346b1ff48c78ac919c80ba58654 | 127.7.39.130:43311 | HEALTHY | <none> | 4 | 0
50d514f034204fc38331aa0a4fd74a1b | 127.7.39.129:37933 | HEALTHY | <none> | 3 | 0
a278448771e14471ab4df1e6649485bd | 127.7.39.131:46461 | HEALTHY | <none> | 4 | 0
e26df472efe547f8bc5951dce352471b | 127.7.39.134:33035 | HEALTHY | <none> | 3 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.7.39.129 | experimental | 127.7.39.129:37933
local_ip_for_outbound_sockets | 127.7.39.130 | experimental | 127.7.39.130:43311
local_ip_for_outbound_sockets | 127.7.39.131 | experimental | 127.7.39.131:46461
local_ip_for_outbound_sockets | 127.7.39.132 | experimental | 127.7.39.132:43977
local_ip_for_outbound_sockets | 127.7.39.133 | experimental | 127.7.39.133:42905
local_ip_for_outbound_sockets | 127.7.39.134 | experimental | 127.7.39.134:33035
local_ip_for_outbound_sockets | 127.7.39.135 | experimental | 127.7.39.135:37645
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.7.39.129:37933
server_dump_info_path | /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.39.130:43311
server_dump_info_path | /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.39.131:46461
server_dump_info_path | /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.7.39.132:43977
server_dump_info_path | /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.39.133:42905
server_dump_info_path | /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.39.134:33035
server_dump_info_path | /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.7.39.135:37645
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.39.148:33673 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 5
Third Quartile | 5
Maximum | 5
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 14:10:12.506165 7326 log_verifier.cc:126] Checking tablet 0dc94f932e9a4319988e700aff042f22
I20240501 14:10:12.634274 7326 log_verifier.cc:177] Verified matching terms for 826 ops in tablet 0dc94f932e9a4319988e700aff042f22
I20240501 14:10:12.634568 7326 log_verifier.cc:126] Checking tablet 341c6b416c374a9ba4fc34d861ba2ef9
I20240501 14:10:12.740625 7326 log_verifier.cc:177] Verified matching terms for 680 ops in tablet 341c6b416c374a9ba4fc34d861ba2ef9
I20240501 14:10:12.740851 7326 log_verifier.cc:126] Checking tablet 3f34112bba8a43a58420148533a6e989
I20240501 14:10:12.873641 7326 log_verifier.cc:177] Verified matching terms for 845 ops in tablet 3f34112bba8a43a58420148533a6e989
I20240501 14:10:12.873904 7326 log_verifier.cc:126] Checking tablet 4d3b3213e1dd426b8efc3fdc733b1842
I20240501 14:10:12.992857 7326 log_verifier.cc:177] Verified matching terms for 789 ops in tablet 4d3b3213e1dd426b8efc3fdc733b1842
I20240501 14:10:12.993072 7326 log_verifier.cc:126] Checking tablet 8a89cc8e4b2b476ba954d3cc64537031
I20240501 14:10:13.121990 7326 log_verifier.cc:177] Verified matching terms for 829 ops in tablet 8a89cc8e4b2b476ba954d3cc64537031
I20240501 14:10:13.122222 7326 log_verifier.cc:126] Checking tablet a067a9d6642349c1b99effece908888c
I20240501 14:10:13.236099 7326 log_verifier.cc:177] Verified matching terms for 736 ops in tablet a067a9d6642349c1b99effece908888c
I20240501 14:10:13.236296 7326 log_verifier.cc:126] Checking tablet b0eb8ac1778a40a0a0e3c6b113ce5741
I20240501 14:10:13.251351 7326 log_verifier.cc:177] Verified matching terms for 2 ops in tablet b0eb8ac1778a40a0a0e3c6b113ce5741
I20240501 14:10:13.251422 7326 log_verifier.cc:126] Checking tablet ba183686332c4ab5b606bbc5f1b4e787
I20240501 14:10:13.387272 7326 log_verifier.cc:177] Verified matching terms for 852 ops in tablet ba183686332c4ab5b606bbc5f1b4e787
I20240501 14:10:13.387513 7326 log_verifier.cc:126] Checking tablet bd836361783c4b3782c0f64c6794488b
I20240501 14:10:13.499850 7326 log_verifier.cc:177] Verified matching terms for 741 ops in tablet bd836361783c4b3782c0f64c6794488b
I20240501 14:10:13.500067 7326 log_verifier.cc:126] Checking tablet bf0b5fd770b8405b9049c52d5d95bc53
I20240501 14:10:13.619181 7326 log_verifier.cc:177] Verified matching terms for 809 ops in tablet bf0b5fd770b8405b9049c52d5d95bc53
I20240501 14:10:13.619428 7326 log_verifier.cc:126] Checking tablet cce2260455ac4edbb9bc54b103c50eb3
I20240501 14:10:13.739955 7326 log_verifier.cc:177] Verified matching terms for 768 ops in tablet cce2260455ac4edbb9bc54b103c50eb3
I20240501 14:10:13.740249 7326 log_verifier.cc:126] Checking tablet cdb343c0875e4709b9637c241dd4d582
I20240501 14:10:13.858170 7326 log_verifier.cc:177] Verified matching terms for 787 ops in tablet cdb343c0875e4709b9637c241dd4d582
I20240501 14:10:13.858408 7326 log_verifier.cc:126] Checking tablet cde568b4e46a4c5499031d0e18ae51de
I20240501 14:10:13.970472 7326 log_verifier.cc:177] Verified matching terms for 730 ops in tablet cde568b4e46a4c5499031d0e18ae51de
I20240501 14:10:13.970710 7326 log_verifier.cc:126] Checking tablet d948c2d5182b415b8aedb4c7b786247f
I20240501 14:10:14.092924 7326 log_verifier.cc:177] Verified matching terms for 818 ops in tablet d948c2d5182b415b8aedb4c7b786247f
I20240501 14:10:14.093175 7326 log_verifier.cc:126] Checking tablet e4266d76760c4f84b16991f8aa38a3c3
I20240501 14:10:14.207687 7326 log_verifier.cc:177] Verified matching terms for 735 ops in tablet e4266d76760c4f84b16991f8aa38a3c3
I20240501 14:10:14.207923 7326 log_verifier.cc:126] Checking tablet f1c30f16a13444cc95132eeb9ea6c96c
I20240501 14:10:14.310676 7326 log_verifier.cc:177] Verified matching terms for 675 ops in tablet f1c30f16a13444cc95132eeb9ea6c96c
I20240501 14:10:14.312965 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.136:0
--local_ip_for_outbound_sockets=127.7.39.136
--webserver_interface=127.7.39.136
--webserver_port=0
--tserver_master_addrs=127.7.39.190:41789
--builtin_ntp_servers=127.7.39.148:33673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 14:10:14.421830 17889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:10:14.422035 17889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:10:14.422092 17889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:10:14.424845 17889 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:10:14.424907 17889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:10:14.424989 17889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.136
I20240501 14:10:14.428445 17889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:33673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.7.39.136
--webserver_port=0
--tserver_master_addrs=127.7.39.190:41789
--never_fsync=true
--heap_profile_path=/tmp/kudu.17889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.136
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:10:14.429242 17889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:10:14.430349 17889 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:10:14.439594 17894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:10:14.441058 17897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:10:14.442004 17895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:10:14.443295 17889 server_base.cc:1034] running on GCE node
I20240501 14:10:14.443647 17889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:10:14.444170 17889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:10:14.445333 17889 hybrid_clock.cc:648] HybridClock initialized: now 1714572614445312 us; error 37 us; skew 500 ppm
I20240501 14:10:14.447290 17889 webserver.cc:460] Webserver started at http://127.7.39.136:37583/ using document root <none> and password file <none>
I20240501 14:10:14.447825 17889 fs_manager.cc:359] Metadata directory not provided
I20240501 14:10:14.447888 17889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:10:14.448063 17889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:10:14.449743 17889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "7379b25903bb465b8da2e85fd48f8a89"
format_stamp: "Formatted at 2024-05-01 14:10:14 on dist-test-slave-rnfw"
I20240501 14:10:14.450166 17889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "7379b25903bb465b8da2e85fd48f8a89"
format_stamp: "Formatted at 2024-05-01 14:10:14 on dist-test-slave-rnfw"
I20240501 14:10:14.454800 17889 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20240501 14:10:14.456844 17903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:14.457747 17889 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:10:14.457836 17889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "7379b25903bb465b8da2e85fd48f8a89"
format_stamp: "Formatted at 2024-05-01 14:10:14 on dist-test-slave-rnfw"
I20240501 14:10:14.457904 17889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:10:14.483177 17889 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:10:14.483811 17889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:10:14.483953 17889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:10:14.484442 17889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:10:14.485245 17889 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:10:14.485291 17889 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:14.485327 17889 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:10:14.485347 17889 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:14.493927 17889 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.136:42889
I20240501 14:10:14.493942 18016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.136:42889 every 8 connection(s)
I20240501 14:10:14.494851 17889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 14:10:14.499935 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 17889
I20240501 14:10:14.500061 7326 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 14:10:14.503535 18017 heartbeater.cc:347] Connected to a master server at 127.7.39.190:41789
I20240501 14:10:14.503746 18017 heartbeater.cc:464] Registering TS with master...
I20240501 14:10:14.504719 18017 heartbeater.cc:511] Master 127.7.39.190:41789 requested a full tablet report, sending...
I20240501 14:10:14.505847 13299 ts_manager.cc:194] Registered new tserver with Master: 7379b25903bb465b8da2e85fd48f8a89 (127.7.39.136:42889)
I20240501 14:10:14.506390 13299 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.136:40649
I20240501 14:10:14.694985 15062 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:14.695199 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:14.698293 13555 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:14.699149 14665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:14.701176 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:14.707649 13687 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:14.716100 17951 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:14.723979 14928 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:14.822520 14665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:14.825903 15062 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:14.827270 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:14.827831 17951 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:14.833724 13687 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:14.847414 13555 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:14.850226 14928 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:14.859020 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:14.902585 14946 consensus_queue.cc:237] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } }
I20240501 14:10:14.904309 13577 raft_consensus.cc:1261] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654 [term 2 FOLLOWER]: Refusing update from remote peer e26df472efe547f8bc5951dce352471b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 14:10:14.905001 18121 consensus_queue.cc:1035] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b [LEADER]: Connected to new peer: Peer: permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240501 14:10:14.908030 14883 consensus_peers.cc:485] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b -> Peer 7379b25903bb465b8da2e85fd48f8a89 (127.7.39.136:42889): Couldn't send request to peer 7379b25903bb465b8da2e85fd48f8a89. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b0eb8ac1778a40a0a0e3c6b113ce5741. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:14.912087 16930 raft_consensus.cc:2932] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER 7379b25903bb465b8da2e85fd48f8a89 (127.7.39.136) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } } }
I20240501 14:10:14.915612 13577 raft_consensus.cc:2932] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER 7379b25903bb465b8da2e85fd48f8a89 (127.7.39.136) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } } }
I20240501 14:10:14.915871 13302 catalog_manager.cc:5484] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b reported cstate change: config changed from index -1 to 3, VOTER 7379b25903bb465b8da2e85fd48f8a89 (127.7.39.136) added. New cstate: current_term: 2 leader_uuid: "e26df472efe547f8bc5951dce352471b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } } }
I20240501 14:10:14.933689 13442 consensus_queue.cc:237] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 852, Committed index: 852, Last appended: 1.852, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 853 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } }
I20240501 14:10:14.935240 14816 raft_consensus.cc:1261] T ba183686332c4ab5b606bbc5f1b4e787 P 4015e9df09424332bc3d7a9bad781641 [term 1 FOLLOWER]: Refusing update from remote peer 50d514f034204fc38331aa0a4fd74a1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 852. Preceding OpId from leader: term: 1 index: 853. (index mismatch)
I20240501 14:10:14.935714 18146 consensus_queue.cc:1035] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 853, Last known committed idx: 852, Time since last communication: 0.000s
I20240501 14:10:14.937594 17088 raft_consensus.cc:2932] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b [term 1 LEADER]: Committing config change with OpId 1.853: config changed from index 206 to 853, VOTER 7379b25903bb465b8da2e85fd48f8a89 (127.7.39.136) added. New config: { opid_index: 853 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } } }
I20240501 14:10:14.938019 14816 raft_consensus.cc:2932] T ba183686332c4ab5b606bbc5f1b4e787 P 4015e9df09424332bc3d7a9bad781641 [term 1 FOLLOWER]: Committing config change with OpId 1.853: config changed from index 206 to 853, VOTER 7379b25903bb465b8da2e85fd48f8a89 (127.7.39.136) added. New config: { opid_index: 853 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } } }
W20240501 14:10:14.938477 13382 consensus_peers.cc:485] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b -> Peer 7379b25903bb465b8da2e85fd48f8a89 (127.7.39.136:42889): Couldn't send request to peer 7379b25903bb465b8da2e85fd48f8a89. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ba183686332c4ab5b606bbc5f1b4e787. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:14.940260 13298 catalog_manager.cc:5484] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b reported cstate change: config changed from index 206 to 853, VOTER 7379b25903bb465b8da2e85fd48f8a89 (127.7.39.136) added. New cstate: current_term: 1 leader_uuid: "50d514f034204fc38331aa0a4fd74a1b" committed_config { opid_index: 853 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } } }
I20240501 14:10:14.958130 13577 consensus_queue.cc:237] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 809, Committed index: 809, Last appended: 1.809, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 810 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } }
I20240501 14:10:14.959559 14816 raft_consensus.cc:1261] T bf0b5fd770b8405b9049c52d5d95bc53 P 4015e9df09424332bc3d7a9bad781641 [term 1 FOLLOWER]: Refusing update from remote peer 43736346b1ff48c78ac919c80ba58654: Log matching property violated. Preceding OpId in replica: term: 1 index: 809. Preceding OpId from leader: term: 1 index: 810. (index mismatch)
I20240501 14:10:14.959991 18166 consensus_queue.cc:1035] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 810, Last known committed idx: 809, Time since last communication: 0.000s
I20240501 14:10:14.961820 16985 raft_consensus.cc:2932] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654 [term 1 LEADER]: Committing config change with OpId 1.810: config changed from index 319 to 810, VOTER 7379b25903bb465b8da2e85fd48f8a89 (127.7.39.136) added. New config: { opid_index: 810 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } } }
W20240501 14:10:14.963088 13513 consensus_peers.cc:485] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654 -> Peer 7379b25903bb465b8da2e85fd48f8a89 (127.7.39.136:42889): Couldn't send request to peer 7379b25903bb465b8da2e85fd48f8a89. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bf0b5fd770b8405b9049c52d5d95bc53. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:14.963824 14816 raft_consensus.cc:2932] T bf0b5fd770b8405b9049c52d5d95bc53 P 4015e9df09424332bc3d7a9bad781641 [term 1 FOLLOWER]: Committing config change with OpId 1.810: config changed from index 319 to 810, VOTER 7379b25903bb465b8da2e85fd48f8a89 (127.7.39.136) added. New config: { opid_index: 810 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } } }
I20240501 14:10:14.964655 13302 catalog_manager.cc:5484] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654 reported cstate change: config changed from index 319 to 810, VOTER 7379b25903bb465b8da2e85fd48f8a89 (127.7.39.136) added. New cstate: current_term: 1 leader_uuid: "43736346b1ff48c78ac919c80ba58654" committed_config { opid_index: 810 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } } }
I20240501 14:10:14.983754 13708 consensus_queue.cc:237] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 736, Committed index: 736, Last appended: 1.736, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 737 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } }
I20240501 14:10:14.985133 14816 raft_consensus.cc:1261] T a067a9d6642349c1b99effece908888c P 4015e9df09424332bc3d7a9bad781641 [term 1 FOLLOWER]: Refusing update from remote peer a278448771e14471ab4df1e6649485bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 736. Preceding OpId from leader: term: 1 index: 737. (index mismatch)
I20240501 14:10:14.985772 18186 consensus_queue.cc:1035] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 737, Last known committed idx: 736, Time since last communication: 0.000s
I20240501 14:10:14.987550 17444 raft_consensus.cc:2932] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd [term 1 LEADER]: Committing config change with OpId 1.737: config changed from index 295 to 737, VOTER 7379b25903bb465b8da2e85fd48f8a89 (127.7.39.136) added. New config: { opid_index: 737 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } } }
W20240501 14:10:14.989015 13644 consensus_peers.cc:485] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd -> Peer 7379b25903bb465b8da2e85fd48f8a89 (127.7.39.136:42889): Couldn't send request to peer 7379b25903bb465b8da2e85fd48f8a89. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a067a9d6642349c1b99effece908888c. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:14.989126 14816 raft_consensus.cc:2932] T a067a9d6642349c1b99effece908888c P 4015e9df09424332bc3d7a9bad781641 [term 1 FOLLOWER]: Committing config change with OpId 1.737: config changed from index 295 to 737, VOTER 7379b25903bb465b8da2e85fd48f8a89 (127.7.39.136) added. New config: { opid_index: 737 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } } }
I20240501 14:10:14.990118 13298 catalog_manager.cc:5484] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd reported cstate change: config changed from index 295 to 737, VOTER 7379b25903bb465b8da2e85fd48f8a89 (127.7.39.136) added. New cstate: current_term: 1 leader_uuid: "a278448771e14471ab4df1e6649485bd" committed_config { opid_index: 737 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } } }
I20240501 14:10:15.051026 15062 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:15.052188 14665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:15.052876 13555 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:15.053094 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:15.059355 14928 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:15.076862 17951 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:15.077061 13687 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:15.087860 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:15.161047 14665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:15.164072 15062 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:15.171116 13687 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:15.171378 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:15.173599 13555 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:15.178599 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:15.190248 14928 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:15.196123 17951 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:15.274171 14665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:15.274818 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:15.275503 15062 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:15.284442 17951 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:15.288950 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:15.291286 13555 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:15.293406 13687 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:15.306566 14928 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:15.348683 18304 ts_tablet_manager.cc:902] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 7379b25903bb465b8da2e85fd48f8a89: Initiating tablet copy from peer e26df472efe547f8bc5951dce352471b (127.7.39.134:33035)
I20240501 14:10:15.349347 18305 ts_tablet_manager.cc:902] T ba183686332c4ab5b606bbc5f1b4e787 P 7379b25903bb465b8da2e85fd48f8a89: Initiating tablet copy from peer 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129:37933)
I20240501 14:10:15.349886 18304 tablet_copy_client.cc:308] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 7379b25903bb465b8da2e85fd48f8a89: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.134:33035
I20240501 14:10:15.350040 18305 tablet_copy_client.cc:308] T ba183686332c4ab5b606bbc5f1b4e787 P 7379b25903bb465b8da2e85fd48f8a89: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.129:37933
I20240501 14:10:15.354712 14968 tablet_copy_service.cc:140] P e26df472efe547f8bc5951dce352471b: Received BeginTabletCopySession request for tablet b0eb8ac1778a40a0a0e3c6b113ce5741 from peer 7379b25903bb465b8da2e85fd48f8a89 ({username='slave'} at 127.7.39.136:41387)
I20240501 14:10:15.354859 14968 tablet_copy_service.cc:161] P e26df472efe547f8bc5951dce352471b: Beginning new tablet copy session on tablet b0eb8ac1778a40a0a0e3c6b113ce5741 from peer 7379b25903bb465b8da2e85fd48f8a89 at {username='slave'} at 127.7.39.136:41387: session id = 7379b25903bb465b8da2e85fd48f8a89-b0eb8ac1778a40a0a0e3c6b113ce5741
I20240501 14:10:15.354852 13466 tablet_copy_service.cc:140] P 50d514f034204fc38331aa0a4fd74a1b: Received BeginTabletCopySession request for tablet ba183686332c4ab5b606bbc5f1b4e787 from peer 7379b25903bb465b8da2e85fd48f8a89 ({username='slave'} at 127.7.39.136:56097)
I20240501 14:10:15.354956 13466 tablet_copy_service.cc:161] P 50d514f034204fc38331aa0a4fd74a1b: Beginning new tablet copy session on tablet ba183686332c4ab5b606bbc5f1b4e787 from peer 7379b25903bb465b8da2e85fd48f8a89 at {username='slave'} at 127.7.39.136:56097: session id = 7379b25903bb465b8da2e85fd48f8a89-ba183686332c4ab5b606bbc5f1b4e787
I20240501 14:10:15.356004 13466 tablet_copy_source_session.cc:205] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:15.356127 14968 tablet_copy_source_session.cc:205] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:15.359239 18304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0eb8ac1778a40a0a0e3c6b113ce5741. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:15.361573 18305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba183686332c4ab5b606bbc5f1b4e787. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:15.367072 18304 tablet_copy_client.cc:783] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 7379b25903bb465b8da2e85fd48f8a89: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:15.367317 18304 tablet_copy_client.cc:647] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 7379b25903bb465b8da2e85fd48f8a89: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:15.369097 18304 tablet_copy_client.cc:523] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 7379b25903bb465b8da2e85fd48f8a89: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:15.369285 18305 tablet_copy_client.cc:783] T ba183686332c4ab5b606bbc5f1b4e787 P 7379b25903bb465b8da2e85fd48f8a89: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:15.369515 18305 tablet_copy_client.cc:647] T ba183686332c4ab5b606bbc5f1b4e787 P 7379b25903bb465b8da2e85fd48f8a89: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:15.371030 18304 tablet_bootstrap.cc:492] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 7379b25903bb465b8da2e85fd48f8a89: Bootstrap starting.
I20240501 14:10:15.374317 18305 tablet_copy_client.cc:523] T ba183686332c4ab5b606bbc5f1b4e787 P 7379b25903bb465b8da2e85fd48f8a89: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:15.375546 18305 tablet_bootstrap.cc:492] T ba183686332c4ab5b606bbc5f1b4e787 P 7379b25903bb465b8da2e85fd48f8a89: Bootstrap starting.
I20240501 14:10:15.380575 18333 ts_tablet_manager.cc:902] T a067a9d6642349c1b99effece908888c P 7379b25903bb465b8da2e85fd48f8a89: Initiating tablet copy from peer a278448771e14471ab4df1e6649485bd (127.7.39.131:46461)
I20240501 14:10:15.382822 18333 tablet_copy_client.cc:308] T a067a9d6642349c1b99effece908888c P 7379b25903bb465b8da2e85fd48f8a89: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.131:46461
I20240501 14:10:15.388542 13728 tablet_copy_service.cc:140] P a278448771e14471ab4df1e6649485bd: Received BeginTabletCopySession request for tablet a067a9d6642349c1b99effece908888c from peer 7379b25903bb465b8da2e85fd48f8a89 ({username='slave'} at 127.7.39.136:59677)
I20240501 14:10:15.388653 13728 tablet_copy_service.cc:161] P a278448771e14471ab4df1e6649485bd: Beginning new tablet copy session on tablet a067a9d6642349c1b99effece908888c from peer 7379b25903bb465b8da2e85fd48f8a89 at {username='slave'} at 127.7.39.136:59677: session id = 7379b25903bb465b8da2e85fd48f8a89-a067a9d6642349c1b99effece908888c
I20240501 14:10:15.389597 13728 tablet_copy_source_session.cc:205] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:15.390561 18333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a067a9d6642349c1b99effece908888c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:15.392683 18333 tablet_copy_client.cc:783] T a067a9d6642349c1b99effece908888c P 7379b25903bb465b8da2e85fd48f8a89: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:15.392844 18333 tablet_copy_client.cc:647] T a067a9d6642349c1b99effece908888c P 7379b25903bb465b8da2e85fd48f8a89: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:15.392894 18304 log.cc:824] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 7379b25903bb465b8da2e85fd48f8a89: Log is configured to *not* fsync() on all Append() calls
I20240501 14:10:15.398794 18333 tablet_copy_client.cc:523] T a067a9d6642349c1b99effece908888c P 7379b25903bb465b8da2e85fd48f8a89: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:15.401823 18333 tablet_bootstrap.cc:492] T a067a9d6642349c1b99effece908888c P 7379b25903bb465b8da2e85fd48f8a89: Bootstrap starting.
I20240501 14:10:15.405886 15062 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:15.406183 14665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:15.412451 18304 tablet_bootstrap.cc:492] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 7379b25903bb465b8da2e85fd48f8a89: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:15.412842 18304 tablet_bootstrap.cc:492] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 7379b25903bb465b8da2e85fd48f8a89: Bootstrap complete.
I20240501 14:10:15.413166 18304 ts_tablet_manager.cc:1367] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 7379b25903bb465b8da2e85fd48f8a89: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.006s
I20240501 14:10:15.420254 18304 raft_consensus.cc:348] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 7379b25903bb465b8da2e85fd48f8a89 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } }
I20240501 14:10:15.420544 18304 raft_consensus.cc:729] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 7379b25903bb465b8da2e85fd48f8a89 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7379b25903bb465b8da2e85fd48f8a89, State: Initialized, Role: FOLLOWER
I20240501 14:10:15.420823 14928 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:15.426980 18304 consensus_queue.cc:260] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 7379b25903bb465b8da2e85fd48f8a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } }
I20240501 14:10:15.428083 18304 ts_tablet_manager.cc:1398] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 7379b25903bb465b8da2e85fd48f8a89: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20240501 14:10:15.431407 18017 heartbeater.cc:503] Master 127.7.39.190:41789 was elected leader, sending a full tablet report...
I20240501 14:10:15.432961 14968 tablet_copy_service.cc:342] P e26df472efe547f8bc5951dce352471b: Request end of tablet copy session 7379b25903bb465b8da2e85fd48f8a89-b0eb8ac1778a40a0a0e3c6b113ce5741 received from {username='slave'} at 127.7.39.136:41387
I20240501 14:10:15.433072 14968 tablet_copy_service.cc:433] P e26df472efe547f8bc5951dce352471b: ending tablet copy session 7379b25903bb465b8da2e85fd48f8a89-b0eb8ac1778a40a0a0e3c6b113ce5741 on tablet b0eb8ac1778a40a0a0e3c6b113ce5741 with peer 7379b25903bb465b8da2e85fd48f8a89
I20240501 14:10:15.441303 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:15.444794 17951 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:15.451341 13687 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:15.452093 13555 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:15.452101 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:15.539199 18304 ts_tablet_manager.cc:902] T bf0b5fd770b8405b9049c52d5d95bc53 P 7379b25903bb465b8da2e85fd48f8a89: Initiating tablet copy from peer 43736346b1ff48c78ac919c80ba58654 (127.7.39.130:43311)
I20240501 14:10:15.539721 18304 tablet_copy_client.cc:308] T bf0b5fd770b8405b9049c52d5d95bc53 P 7379b25903bb465b8da2e85fd48f8a89: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.130:43311
I20240501 14:10:15.543995 13597 tablet_copy_service.cc:140] P 43736346b1ff48c78ac919c80ba58654: Received BeginTabletCopySession request for tablet bf0b5fd770b8405b9049c52d5d95bc53 from peer 7379b25903bb465b8da2e85fd48f8a89 ({username='slave'} at 127.7.39.136:53765)
I20240501 14:10:15.544113 13597 tablet_copy_service.cc:161] P 43736346b1ff48c78ac919c80ba58654: Beginning new tablet copy session on tablet bf0b5fd770b8405b9049c52d5d95bc53 from peer 7379b25903bb465b8da2e85fd48f8a89 at {username='slave'} at 127.7.39.136:53765: session id = 7379b25903bb465b8da2e85fd48f8a89-bf0b5fd770b8405b9049c52d5d95bc53
I20240501 14:10:15.545099 13597 tablet_copy_source_session.cc:205] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:15.545933 18304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf0b5fd770b8405b9049c52d5d95bc53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:15.548329 18304 tablet_copy_client.cc:783] T bf0b5fd770b8405b9049c52d5d95bc53 P 7379b25903bb465b8da2e85fd48f8a89: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:15.548534 18304 tablet_copy_client.cc:647] T bf0b5fd770b8405b9049c52d5d95bc53 P 7379b25903bb465b8da2e85fd48f8a89: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:15.551625 18304 tablet_copy_client.cc:523] T bf0b5fd770b8405b9049c52d5d95bc53 P 7379b25903bb465b8da2e85fd48f8a89: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:15.552983 18304 tablet_bootstrap.cc:492] T bf0b5fd770b8405b9049c52d5d95bc53 P 7379b25903bb465b8da2e85fd48f8a89: Bootstrap starting.
I20240501 14:10:15.617874 15062 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:15.637102 13555 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:15.637420 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:15.648630 13687 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:15.659596 14665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:15.660584 14928 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:15.663184 17951 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:15.698863 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:15.850715 14665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:15.870600 13555 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:15.873590 15062 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:15.876927 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:15.877985 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:15.880982 17951 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:15.885282 17971 raft_consensus.cc:1203] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 7379b25903bb465b8da2e85fd48f8a89 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20240501 14:10:15.912654 13687 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:15.920810 14928 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:16.059194 18333 tablet_bootstrap.cc:492] T a067a9d6642349c1b99effece908888c P 7379b25903bb465b8da2e85fd48f8a89: Bootstrap replayed 1/1 log segments. Stats: ops{read=737 overwritten=0 applied=737 ignored=0} inserts{seen=1420 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:16.059291 13555 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:16.059531 18333 tablet_bootstrap.cc:492] T a067a9d6642349c1b99effece908888c P 7379b25903bb465b8da2e85fd48f8a89: Bootstrap complete.
I20240501 14:10:16.059849 18333 ts_tablet_manager.cc:1367] T a067a9d6642349c1b99effece908888c P 7379b25903bb465b8da2e85fd48f8a89: Time spent bootstrapping tablet: real 0.658s user 0.472s sys 0.039s
I20240501 14:10:16.060505 18333 raft_consensus.cc:348] T a067a9d6642349c1b99effece908888c P 7379b25903bb465b8da2e85fd48f8a89 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 737 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } }
I20240501 14:10:16.060667 18333 raft_consensus.cc:729] T a067a9d6642349c1b99effece908888c P 7379b25903bb465b8da2e85fd48f8a89 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7379b25903bb465b8da2e85fd48f8a89, State: Initialized, Role: FOLLOWER
I20240501 14:10:16.060899 18333 consensus_queue.cc:260] T a067a9d6642349c1b99effece908888c P 7379b25903bb465b8da2e85fd48f8a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 737, Last appended: 1.737, Last appended by leader: 737, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 737 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } }
I20240501 14:10:16.061971 18333 ts_tablet_manager.cc:1398] T a067a9d6642349c1b99effece908888c P 7379b25903bb465b8da2e85fd48f8a89: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:10:16.067490 13728 tablet_copy_service.cc:342] P a278448771e14471ab4df1e6649485bd: Request end of tablet copy session 7379b25903bb465b8da2e85fd48f8a89-a067a9d6642349c1b99effece908888c received from {username='slave'} at 127.7.39.136:59677
I20240501 14:10:16.067572 13728 tablet_copy_service.cc:433] P a278448771e14471ab4df1e6649485bd: ending tablet copy session 7379b25903bb465b8da2e85fd48f8a89-a067a9d6642349c1b99effece908888c on tablet a067a9d6642349c1b99effece908888c with peer 7379b25903bb465b8da2e85fd48f8a89
I20240501 14:10:16.072152 14928 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:16.077230 13687 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:16.084743 17951 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:16.096352 14665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:16.097975 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:16.099323 15062 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:16.099802 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:16.175936 18305 tablet_bootstrap.cc:492] T ba183686332c4ab5b606bbc5f1b4e787 P 7379b25903bb465b8da2e85fd48f8a89: Bootstrap replayed 1/1 log segments. Stats: ops{read=853 overwritten=0 applied=853 ignored=0} inserts{seen=1536 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:16.176402 18305 tablet_bootstrap.cc:492] T ba183686332c4ab5b606bbc5f1b4e787 P 7379b25903bb465b8da2e85fd48f8a89: Bootstrap complete.
I20240501 14:10:16.176766 18305 ts_tablet_manager.cc:1367] T ba183686332c4ab5b606bbc5f1b4e787 P 7379b25903bb465b8da2e85fd48f8a89: Time spent bootstrapping tablet: real 0.801s user 0.537s sys 0.058s
I20240501 14:10:16.177263 18305 raft_consensus.cc:348] T ba183686332c4ab5b606bbc5f1b4e787 P 7379b25903bb465b8da2e85fd48f8a89 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 853 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } }
I20240501 14:10:16.177687 18305 raft_consensus.cc:729] T ba183686332c4ab5b606bbc5f1b4e787 P 7379b25903bb465b8da2e85fd48f8a89 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7379b25903bb465b8da2e85fd48f8a89, State: Initialized, Role: FOLLOWER
I20240501 14:10:16.177891 18305 consensus_queue.cc:260] T ba183686332c4ab5b606bbc5f1b4e787 P 7379b25903bb465b8da2e85fd48f8a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 853, Last appended: 1.853, Last appended by leader: 853, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 853 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } }
I20240501 14:10:16.178856 18305 ts_tablet_manager.cc:1398] T ba183686332c4ab5b606bbc5f1b4e787 P 7379b25903bb465b8da2e85fd48f8a89: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:10:16.179445 13466 tablet_copy_service.cc:342] P 50d514f034204fc38331aa0a4fd74a1b: Request end of tablet copy session 7379b25903bb465b8da2e85fd48f8a89-ba183686332c4ab5b606bbc5f1b4e787 received from {username='slave'} at 127.7.39.136:56097
I20240501 14:10:16.179540 13466 tablet_copy_service.cc:433] P 50d514f034204fc38331aa0a4fd74a1b: ending tablet copy session 7379b25903bb465b8da2e85fd48f8a89-ba183686332c4ab5b606bbc5f1b4e787 on tablet ba183686332c4ab5b606bbc5f1b4e787 with peer 7379b25903bb465b8da2e85fd48f8a89
I20240501 14:10:16.207960 17971 raft_consensus.cc:1203] T ba183686332c4ab5b606bbc5f1b4e787 P 7379b25903bb465b8da2e85fd48f8a89 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.852->[1.853-1.853] Dedup: 1.853->[]
I20240501 14:10:16.233238 15062 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:16.240926 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:16.245085 13555 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:16.245797 14665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:16.251659 13687 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:16.266461 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:16.276398 17951 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:16.276868 14928 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:16.311439 18304 tablet_bootstrap.cc:492] T bf0b5fd770b8405b9049c52d5d95bc53 P 7379b25903bb465b8da2e85fd48f8a89: Bootstrap replayed 1/1 log segments. Stats: ops{read=810 overwritten=0 applied=810 ignored=0} inserts{seen=1493 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:16.311842 18304 tablet_bootstrap.cc:492] T bf0b5fd770b8405b9049c52d5d95bc53 P 7379b25903bb465b8da2e85fd48f8a89: Bootstrap complete.
I20240501 14:10:16.312134 18304 ts_tablet_manager.cc:1367] T bf0b5fd770b8405b9049c52d5d95bc53 P 7379b25903bb465b8da2e85fd48f8a89: Time spent bootstrapping tablet: real 0.759s user 0.540s sys 0.043s
I20240501 14:10:16.312855 18304 raft_consensus.cc:348] T bf0b5fd770b8405b9049c52d5d95bc53 P 7379b25903bb465b8da2e85fd48f8a89 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 810 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } }
I20240501 14:10:16.313001 18304 raft_consensus.cc:729] T bf0b5fd770b8405b9049c52d5d95bc53 P 7379b25903bb465b8da2e85fd48f8a89 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7379b25903bb465b8da2e85fd48f8a89, State: Initialized, Role: FOLLOWER
I20240501 14:10:16.313146 18304 consensus_queue.cc:260] T bf0b5fd770b8405b9049c52d5d95bc53 P 7379b25903bb465b8da2e85fd48f8a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 810, Last appended: 1.810, Last appended by leader: 810, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 810 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } }
I20240501 14:10:16.313426 18304 ts_tablet_manager.cc:1398] T bf0b5fd770b8405b9049c52d5d95bc53 P 7379b25903bb465b8da2e85fd48f8a89: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:16.314237 13597 tablet_copy_service.cc:342] P 43736346b1ff48c78ac919c80ba58654: Request end of tablet copy session 7379b25903bb465b8da2e85fd48f8a89-bf0b5fd770b8405b9049c52d5d95bc53 received from {username='slave'} at 127.7.39.136:53765
I20240501 14:10:16.314321 13597 tablet_copy_service.cc:433] P 43736346b1ff48c78ac919c80ba58654: ending tablet copy session 7379b25903bb465b8da2e85fd48f8a89-bf0b5fd770b8405b9049c52d5d95bc53 on tablet bf0b5fd770b8405b9049c52d5d95bc53 with peer 7379b25903bb465b8da2e85fd48f8a89
I20240501 14:10:16.326859 17971 raft_consensus.cc:1203] T a067a9d6642349c1b99effece908888c P 7379b25903bb465b8da2e85fd48f8a89 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.736->[1.737-1.737] Dedup: 1.737->[]
I20240501 14:10:16.597343 14665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:16.597645 15062 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:16.599267 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:16.609711 13555 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:16.611678 13687 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:16.613376 14928 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:16.623961 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:16.625129 17951 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:16.656093 13708 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a067a9d6642349c1b99effece908888c"
dest_uuid: "a278448771e14471ab4df1e6649485bd"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57728
I20240501 14:10:16.656253 13708 raft_consensus.cc:595] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd [term 1 LEADER]: Received request to transfer leadership
I20240501 14:10:16.665143 17971 raft_consensus.cc:1203] T bf0b5fd770b8405b9049c52d5d95bc53 P 7379b25903bb465b8da2e85fd48f8a89 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.809->[1.810-1.810] Dedup: 1.810->[]
I20240501 14:10:16.709321 14665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:16.711506 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:16.719373 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:16.720875 13687 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:16.721577 13555 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:16.722805 15062 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:16.723186 17951 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:16.756024 14928 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:16.778721 18592 raft_consensus.cc:979] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd: : Instructing follower 7379b25903bb465b8da2e85fd48f8a89 to start an election
I20240501 14:10:16.778843 18592 raft_consensus.cc:1067] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd [term 1 LEADER]: Signalling peer 7379b25903bb465b8da2e85fd48f8a89 to start an election
I20240501 14:10:16.779309 17971 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a067a9d6642349c1b99effece908888c"
dest_uuid: "7379b25903bb465b8da2e85fd48f8a89"
from {username='slave'} at 127.7.39.131:46981
I20240501 14:10:16.779518 17971 raft_consensus.cc:482] T a067a9d6642349c1b99effece908888c P 7379b25903bb465b8da2e85fd48f8a89 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:10:16.779666 17971 raft_consensus.cc:3037] T a067a9d6642349c1b99effece908888c P 7379b25903bb465b8da2e85fd48f8a89 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:16.780643 17971 raft_consensus.cc:504] T a067a9d6642349c1b99effece908888c P 7379b25903bb465b8da2e85fd48f8a89 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 737 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } }
I20240501 14:10:16.781797 17971 leader_election.cc:288] T a067a9d6642349c1b99effece908888c P 7379b25903bb465b8da2e85fd48f8a89 [CANDIDATE]: Term 2 election: Requested vote from peers a278448771e14471ab4df1e6649485bd (127.7.39.131:46461), 4015e9df09424332bc3d7a9bad781641 (127.7.39.133:42905)
I20240501 14:10:16.782246 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a067a9d6642349c1b99effece908888c" candidate_uuid: "7379b25903bb465b8da2e85fd48f8a89" candidate_term: 2 candidate_status { last_received { term: 1 index: 737 } } ignore_live_leader: true dest_uuid: "a278448771e14471ab4df1e6649485bd"
I20240501 14:10:16.782374 13708 raft_consensus.cc:3032] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:10:16.782414 13708 raft_consensus.cc:729] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a278448771e14471ab4df1e6649485bd, State: Running, Role: LEADER
I20240501 14:10:16.782564 13708 consensus_queue.cc:260] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 737, Committed index: 737, Last appended: 1.737, Last appended by leader: 737, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 737 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } }
I20240501 14:10:16.782748 13708 raft_consensus.cc:3037] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:16.783771 13708 raft_consensus.cc:2454] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7379b25903bb465b8da2e85fd48f8a89 in term 2.
I20240501 14:10:16.784160 17904 leader_election.cc:302] T a067a9d6642349c1b99effece908888c P 7379b25903bb465b8da2e85fd48f8a89 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7379b25903bb465b8da2e85fd48f8a89, a278448771e14471ab4df1e6649485bd; no voters:
I20240501 14:10:16.784427 18350 raft_consensus.cc:2781] T a067a9d6642349c1b99effece908888c P 7379b25903bb465b8da2e85fd48f8a89 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:10:16.784646 18350 raft_consensus.cc:686] T a067a9d6642349c1b99effece908888c P 7379b25903bb465b8da2e85fd48f8a89 [term 2 LEADER]: Becoming Leader. State: Replica: 7379b25903bb465b8da2e85fd48f8a89, State: Running, Role: LEADER
I20240501 14:10:16.784950 18350 consensus_queue.cc:237] T a067a9d6642349c1b99effece908888c P 7379b25903bb465b8da2e85fd48f8a89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 737, Committed index: 737, Last appended: 1.737, Last appended by leader: 737, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 737 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } }
I20240501 14:10:16.786314 14816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a067a9d6642349c1b99effece908888c" candidate_uuid: "7379b25903bb465b8da2e85fd48f8a89" candidate_term: 2 candidate_status { last_received { term: 1 index: 737 } } ignore_live_leader: true dest_uuid: "4015e9df09424332bc3d7a9bad781641"
I20240501 14:10:16.786487 14816 raft_consensus.cc:3037] T a067a9d6642349c1b99effece908888c P 4015e9df09424332bc3d7a9bad781641 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:16.787564 14816 raft_consensus.cc:2454] T a067a9d6642349c1b99effece908888c P 4015e9df09424332bc3d7a9bad781641 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7379b25903bb465b8da2e85fd48f8a89 in term 2.
I20240501 14:10:16.787722 13295 catalog_manager.cc:5484] T a067a9d6642349c1b99effece908888c P 7379b25903bb465b8da2e85fd48f8a89 reported cstate change: term changed from 1 to 2, leader changed from a278448771e14471ab4df1e6649485bd (127.7.39.131) to 7379b25903bb465b8da2e85fd48f8a89 (127.7.39.136). New cstate: current_term: 2 leader_uuid: "7379b25903bb465b8da2e85fd48f8a89" committed_config { opid_index: 737 OBSOLETE_local: false peers { permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } } }
I20240501 14:10:16.836077 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:16.837630 14665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:16.839552 15062 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:16.846275 13687 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:16.856724 17951 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:16.858502 14928 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:16.864059 13555 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:16.865374 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:16.900341 13577 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bf0b5fd770b8405b9049c52d5d95bc53"
dest_uuid: "43736346b1ff48c78ac919c80ba58654"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60728
I20240501 14:10:16.900511 13577 raft_consensus.cc:595] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:10:16.954435 14665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:16.960105 13555 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:16.960371 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:16.963361 15062 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:16.965700 14928 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:16.971397 17951 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:16.987622 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:16.991577 13687 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:17.071489 14796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:17.072336 15062 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:17.076885 14665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:17.077275 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:17.079891 17951 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:17.083277 13555 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:17.083431 13687 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:17.086066 14928 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:17.095285 18169 raft_consensus.cc:979] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654: : Instructing follower 4015e9df09424332bc3d7a9bad781641 to start an election
I20240501 14:10:17.095399 18169 raft_consensus.cc:1067] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654 [term 1 LEADER]: Signalling peer 4015e9df09424332bc3d7a9bad781641 to start an election
I20240501 14:10:17.099385 14816 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bf0b5fd770b8405b9049c52d5d95bc53"
dest_uuid: "4015e9df09424332bc3d7a9bad781641"
from {username='slave'} at 127.7.39.130:41115
I20240501 14:10:17.099617 14816 raft_consensus.cc:482] T bf0b5fd770b8405b9049c52d5d95bc53 P 4015e9df09424332bc3d7a9bad781641 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:10:17.099682 14816 raft_consensus.cc:3037] T bf0b5fd770b8405b9049c52d5d95bc53 P 4015e9df09424332bc3d7a9bad781641 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:17.100883 14816 raft_consensus.cc:504] T bf0b5fd770b8405b9049c52d5d95bc53 P 4015e9df09424332bc3d7a9bad781641 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 810 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } }
I20240501 14:10:17.119285 14816 leader_election.cc:288] T bf0b5fd770b8405b9049c52d5d95bc53 P 4015e9df09424332bc3d7a9bad781641 [CANDIDATE]: Term 2 election: Requested vote from peers 43736346b1ff48c78ac919c80ba58654 (127.7.39.130:43311), 7379b25903bb465b8da2e85fd48f8a89 (127.7.39.136:42889)
I20240501 14:10:17.119834 13577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0b5fd770b8405b9049c52d5d95bc53" candidate_uuid: "4015e9df09424332bc3d7a9bad781641" candidate_term: 2 candidate_status { last_received { term: 1 index: 810 } } ignore_live_leader: true dest_uuid: "43736346b1ff48c78ac919c80ba58654"
I20240501 14:10:17.120005 13577 raft_consensus.cc:3032] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:10:17.120079 13577 raft_consensus.cc:729] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 43736346b1ff48c78ac919c80ba58654, State: Running, Role: LEADER
I20240501 14:10:17.120241 13577 consensus_queue.cc:260] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 810, Committed index: 810, Last appended: 1.810, Last appended by leader: 810, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 810 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } }
I20240501 14:10:17.120430 13577 raft_consensus.cc:3037] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:17.121465 13577 raft_consensus.cc:2454] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4015e9df09424332bc3d7a9bad781641 in term 2.
I20240501 14:10:17.121862 14752 leader_election.cc:302] T bf0b5fd770b8405b9049c52d5d95bc53 P 4015e9df09424332bc3d7a9bad781641 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4015e9df09424332bc3d7a9bad781641, 43736346b1ff48c78ac919c80ba58654; no voters:
I20240501 14:10:17.122224 18710 raft_consensus.cc:2781] T bf0b5fd770b8405b9049c52d5d95bc53 P 4015e9df09424332bc3d7a9bad781641 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:10:17.122395 18710 raft_consensus.cc:686] T bf0b5fd770b8405b9049c52d5d95bc53 P 4015e9df09424332bc3d7a9bad781641 [term 2 LEADER]: Becoming Leader. State: Replica: 4015e9df09424332bc3d7a9bad781641, State: Running, Role: LEADER
I20240501 14:10:17.122680 18710 consensus_queue.cc:237] T bf0b5fd770b8405b9049c52d5d95bc53 P 4015e9df09424332bc3d7a9bad781641 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 810, Committed index: 810, Last appended: 1.810, Last appended by leader: 810, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 810 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } }
I20240501 14:10:17.125015 13304 catalog_manager.cc:5484] T bf0b5fd770b8405b9049c52d5d95bc53 P 4015e9df09424332bc3d7a9bad781641 reported cstate change: term changed from 1 to 2, leader changed from 43736346b1ff48c78ac919c80ba58654 (127.7.39.130) to 4015e9df09424332bc3d7a9bad781641 (127.7.39.133). New cstate: current_term: 2 leader_uuid: "4015e9df09424332bc3d7a9bad781641" committed_config { opid_index: 810 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } } }
I20240501 14:10:17.128472 17971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0b5fd770b8405b9049c52d5d95bc53" candidate_uuid: "4015e9df09424332bc3d7a9bad781641" candidate_term: 2 candidate_status { last_received { term: 1 index: 810 } } ignore_live_leader: true dest_uuid: "7379b25903bb465b8da2e85fd48f8a89"
I20240501 14:10:17.128651 17971 raft_consensus.cc:3037] T bf0b5fd770b8405b9049c52d5d95bc53 P 7379b25903bb465b8da2e85fd48f8a89 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:17.129700 17971 raft_consensus.cc:2454] T bf0b5fd770b8405b9049c52d5d95bc53 P 7379b25903bb465b8da2e85fd48f8a89 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4015e9df09424332bc3d7a9bad781641 in term 2.
I20240501 14:10:17.137028 14946 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b0eb8ac1778a40a0a0e3c6b113ce5741"
dest_uuid: "e26df472efe547f8bc5951dce352471b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49190
I20240501 14:10:17.137171 14946 raft_consensus.cc:595] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b [term 2 LEADER]: Received request to transfer leadership
I20240501 14:10:17.187443 15062 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:17.190809 14665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:17.199618 14796 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:17.200212 13687 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:17.200763 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:17.204203 13555 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:17.209815 14928 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:17.215505 17951 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:17.277832 13708 raft_consensus.cc:1261] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd [term 2 FOLLOWER]: Refusing update from remote peer 7379b25903bb465b8da2e85fd48f8a89: Log matching property violated. Preceding OpId in replica: term: 1 index: 737. Preceding OpId from leader: term: 2 index: 738. (index mismatch)
I20240501 14:10:17.278533 18350 consensus_queue.cc:1035] T a067a9d6642349c1b99effece908888c P 7379b25903bb465b8da2e85fd48f8a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a278448771e14471ab4df1e6649485bd" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 46461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 738, Last known committed idx: 737, Time since last communication: 0.000s
I20240501 14:10:17.282142 14816 raft_consensus.cc:1261] T a067a9d6642349c1b99effece908888c P 4015e9df09424332bc3d7a9bad781641 [term 2 FOLLOWER]: Refusing update from remote peer 7379b25903bb465b8da2e85fd48f8a89: Log matching property violated. Preceding OpId in replica: term: 1 index: 737. Preceding OpId from leader: term: 2 index: 738. (index mismatch)
I20240501 14:10:17.282548 18350 consensus_queue.cc:1035] T a067a9d6642349c1b99effece908888c P 7379b25903bb465b8da2e85fd48f8a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 738, Last known committed idx: 737, Time since last communication: 0.000s
I20240501 14:10:17.288468 18474 raft_consensus.cc:979] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b: : Instructing follower 43736346b1ff48c78ac919c80ba58654 to start an election
I20240501 14:10:17.288586 18474 raft_consensus.cc:1067] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b [term 2 LEADER]: Signalling peer 43736346b1ff48c78ac919c80ba58654 to start an election
I20240501 14:10:17.289566 13577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b0eb8ac1778a40a0a0e3c6b113ce5741"
dest_uuid: "43736346b1ff48c78ac919c80ba58654"
from {username='slave'} at 127.7.39.134:45565
I20240501 14:10:17.289723 13577 raft_consensus.cc:482] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:10:17.289800 13577 raft_consensus.cc:3037] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:10:17.290983 13577 raft_consensus.cc:504] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } }
I20240501 14:10:17.291507 13577 leader_election.cc:288] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654 [CANDIDATE]: Term 3 election: Requested vote from peers e26df472efe547f8bc5951dce352471b (127.7.39.134:33035), 7379b25903bb465b8da2e85fd48f8a89 (127.7.39.136:42889)
I20240501 14:10:17.292552 14946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0eb8ac1778a40a0a0e3c6b113ce5741" candidate_uuid: "43736346b1ff48c78ac919c80ba58654" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "e26df472efe547f8bc5951dce352471b"
I20240501 14:10:17.292565 17971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0eb8ac1778a40a0a0e3c6b113ce5741" candidate_uuid: "43736346b1ff48c78ac919c80ba58654" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "7379b25903bb465b8da2e85fd48f8a89"
I20240501 14:10:17.292678 14946 raft_consensus.cc:3032] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:10:17.292698 17971 raft_consensus.cc:3037] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 7379b25903bb465b8da2e85fd48f8a89 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:10:17.292722 14946 raft_consensus.cc:729] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e26df472efe547f8bc5951dce352471b, State: Running, Role: LEADER
I20240501 14:10:17.292868 14946 consensus_queue.cc:260] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } }
I20240501 14:10:17.293059 14946 raft_consensus.cc:3037] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:10:17.293769 17971 raft_consensus.cc:2454] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 7379b25903bb465b8da2e85fd48f8a89 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43736346b1ff48c78ac919c80ba58654 in term 3.
I20240501 14:10:17.293990 14946 raft_consensus.cc:2454] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43736346b1ff48c78ac919c80ba58654 in term 3.
I20240501 14:10:17.294128 13513 leader_election.cc:302] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43736346b1ff48c78ac919c80ba58654, 7379b25903bb465b8da2e85fd48f8a89; no voters:
I20240501 14:10:17.294338 18559 raft_consensus.cc:2781] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:10:17.294494 18559 raft_consensus.cc:686] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654 [term 3 LEADER]: Becoming Leader. State: Replica: 43736346b1ff48c78ac919c80ba58654, State: Running, Role: LEADER
I20240501 14:10:17.294689 18559 consensus_queue.cc:237] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } }
I20240501 14:10:17.296370 13302 catalog_manager.cc:5484] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654 reported cstate change: term changed from 2 to 3, leader changed from e26df472efe547f8bc5951dce352471b (127.7.39.134) to 43736346b1ff48c78ac919c80ba58654 (127.7.39.130). New cstate: current_term: 3 leader_uuid: "43736346b1ff48c78ac919c80ba58654" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 } } peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } } }
I20240501 14:10:17.313975 15062 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:17.319507 14665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:17.320525 17951 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:17.321033 13555 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:17.321142 14796 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:17.322495 13687 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:17.333110 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:17.345708 14928 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:17.381304 13442 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ba183686332c4ab5b606bbc5f1b4e787"
dest_uuid: "50d514f034204fc38331aa0a4fd74a1b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45638
I20240501 14:10:17.381516 13442 raft_consensus.cc:595] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b [term 1 LEADER]: Received request to transfer leadership
I20240501 14:10:17.450842 15062 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:17.453964 14665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:17.454152 13555 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:17.466027 14796 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:17.470499 13426 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:17.472246 17951 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:17.473085 13687 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:17.483604 14928 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:17.583343 17088 raft_consensus.cc:979] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b: : Instructing follower 7379b25903bb465b8da2e85fd48f8a89 to start an election
I20240501 14:10:17.583477 18836 raft_consensus.cc:1067] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b [term 1 LEADER]: Signalling peer 7379b25903bb465b8da2e85fd48f8a89 to start an election
I20240501 14:10:17.584095 17971 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ba183686332c4ab5b606bbc5f1b4e787"
dest_uuid: "7379b25903bb465b8da2e85fd48f8a89"
from {username='slave'} at 127.7.39.129:50127
I20240501 14:10:17.584283 17971 raft_consensus.cc:482] T ba183686332c4ab5b606bbc5f1b4e787 P 7379b25903bb465b8da2e85fd48f8a89 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:10:17.584355 17971 raft_consensus.cc:3037] T ba183686332c4ab5b606bbc5f1b4e787 P 7379b25903bb465b8da2e85fd48f8a89 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:17.585251 17971 raft_consensus.cc:504] T ba183686332c4ab5b606bbc5f1b4e787 P 7379b25903bb465b8da2e85fd48f8a89 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 853 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } }
I20240501 14:10:17.585712 17971 leader_election.cc:288] T ba183686332c4ab5b606bbc5f1b4e787 P 7379b25903bb465b8da2e85fd48f8a89 [CANDIDATE]: Term 2 election: Requested vote from peers 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129:37933), 4015e9df09424332bc3d7a9bad781641 (127.7.39.133:42905)
I20240501 14:10:17.586031 13442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba183686332c4ab5b606bbc5f1b4e787" candidate_uuid: "7379b25903bb465b8da2e85fd48f8a89" candidate_term: 2 candidate_status { last_received { term: 1 index: 853 } } ignore_live_leader: true dest_uuid: "50d514f034204fc38331aa0a4fd74a1b"
I20240501 14:10:17.586156 13442 raft_consensus.cc:3032] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:10:17.586202 13442 raft_consensus.cc:729] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 50d514f034204fc38331aa0a4fd74a1b, State: Running, Role: LEADER
I20240501 14:10:17.586163 14816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba183686332c4ab5b606bbc5f1b4e787" candidate_uuid: "7379b25903bb465b8da2e85fd48f8a89" candidate_term: 2 candidate_status { last_received { term: 1 index: 853 } } ignore_live_leader: true dest_uuid: "4015e9df09424332bc3d7a9bad781641"
I20240501 14:10:17.586310 14816 raft_consensus.cc:3037] T ba183686332c4ab5b606bbc5f1b4e787 P 4015e9df09424332bc3d7a9bad781641 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:17.586364 13442 consensus_queue.cc:260] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 853, Committed index: 853, Last appended: 1.853, Last appended by leader: 853, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 853 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } }
I20240501 14:10:17.586580 13442 raft_consensus.cc:3037] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:17.587440 14816 raft_consensus.cc:2454] T ba183686332c4ab5b606bbc5f1b4e787 P 4015e9df09424332bc3d7a9bad781641 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7379b25903bb465b8da2e85fd48f8a89 in term 2.
I20240501 14:10:17.587713 13442 raft_consensus.cc:2454] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7379b25903bb465b8da2e85fd48f8a89 in term 2.
I20240501 14:10:17.587841 17905 leader_election.cc:302] T ba183686332c4ab5b606bbc5f1b4e787 P 7379b25903bb465b8da2e85fd48f8a89 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4015e9df09424332bc3d7a9bad781641, 7379b25903bb465b8da2e85fd48f8a89; no voters:
I20240501 14:10:17.588088 18350 raft_consensus.cc:2781] T ba183686332c4ab5b606bbc5f1b4e787 P 7379b25903bb465b8da2e85fd48f8a89 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:10:17.588181 18350 raft_consensus.cc:686] T ba183686332c4ab5b606bbc5f1b4e787 P 7379b25903bb465b8da2e85fd48f8a89 [term 2 LEADER]: Becoming Leader. State: Replica: 7379b25903bb465b8da2e85fd48f8a89, State: Running, Role: LEADER
I20240501 14:10:17.588325 18350 consensus_queue.cc:237] T ba183686332c4ab5b606bbc5f1b4e787 P 7379b25903bb465b8da2e85fd48f8a89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 853, Committed index: 853, Last appended: 1.853, Last appended by leader: 853, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 853 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } }
I20240501 14:10:17.590387 13300 catalog_manager.cc:5484] T ba183686332c4ab5b606bbc5f1b4e787 P 7379b25903bb465b8da2e85fd48f8a89 reported cstate change: term changed from 1 to 2, leader changed from 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129) to 7379b25903bb465b8da2e85fd48f8a89 (127.7.39.136). New cstate: current_term: 2 leader_uuid: "7379b25903bb465b8da2e85fd48f8a89" committed_config { opid_index: 853 OBSOLETE_local: false peers { permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 } } peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } } }
I20240501 14:10:17.694505 14946 raft_consensus.cc:1261] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b [term 3 FOLLOWER]: Refusing update from remote peer 43736346b1ff48c78ac919c80ba58654: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240501 14:10:17.694990 18559 consensus_queue.cc:1035] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e26df472efe547f8bc5951dce352471b" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 33035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:17.697598 17971 raft_consensus.cc:1261] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 7379b25903bb465b8da2e85fd48f8a89 [term 3 FOLLOWER]: Refusing update from remote peer 43736346b1ff48c78ac919c80ba58654: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240501 14:10:17.698091 18559 consensus_queue.cc:1035] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:17.717015 17971 raft_consensus.cc:1261] T bf0b5fd770b8405b9049c52d5d95bc53 P 7379b25903bb465b8da2e85fd48f8a89 [term 2 FOLLOWER]: Refusing update from remote peer 4015e9df09424332bc3d7a9bad781641: Log matching property violated. Preceding OpId in replica: term: 1 index: 810. Preceding OpId from leader: term: 2 index: 811. (index mismatch)
I20240501 14:10:17.717504 18710 consensus_queue.cc:1035] T bf0b5fd770b8405b9049c52d5d95bc53 P 4015e9df09424332bc3d7a9bad781641 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 811, Last known committed idx: 810, Time since last communication: 0.000s
I20240501 14:10:17.720337 13577 raft_consensus.cc:1261] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654 [term 2 FOLLOWER]: Refusing update from remote peer 4015e9df09424332bc3d7a9bad781641: Log matching property violated. Preceding OpId in replica: term: 1 index: 810. Preceding OpId from leader: term: 2 index: 811. (index mismatch)
I20240501 14:10:17.720788 18710 consensus_queue.cc:1035] T bf0b5fd770b8405b9049c52d5d95bc53 P 4015e9df09424332bc3d7a9bad781641 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 811, Last known committed idx: 810, Time since last communication: 0.000s
I20240501 14:10:17.791958 14665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:17.794274 13555 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:17.795866 17951 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:17.796432 15062 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:17.797751 13426 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:17.809929 14796 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:17.810765 14928 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:17.814406 13687 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:17.866145 17971 consensus_queue.cc:237] T a067a9d6642349c1b99effece908888c P 7379b25903bb465b8da2e85fd48f8a89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 738, Committed index: 738, Last appended: 2.738, Last appended by leader: 737, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 739 OBSOLETE_local: false peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } }
I20240501 14:10:17.867322 14816 raft_consensus.cc:1261] T a067a9d6642349c1b99effece908888c P 4015e9df09424332bc3d7a9bad781641 [term 2 FOLLOWER]: Refusing update from remote peer 7379b25903bb465b8da2e85fd48f8a89: Log matching property violated. Preceding OpId in replica: term: 2 index: 738. Preceding OpId from leader: term: 2 index: 739. (index mismatch)
I20240501 14:10:17.867718 18600 consensus_queue.cc:1035] T a067a9d6642349c1b99effece908888c P 7379b25903bb465b8da2e85fd48f8a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 739, Last known committed idx: 738, Time since last communication: 0.000s
I20240501 14:10:17.869501 18350 raft_consensus.cc:2932] T a067a9d6642349c1b99effece908888c P 7379b25903bb465b8da2e85fd48f8a89 [term 2 LEADER]: Committing config change with OpId 2.739: config changed from index 737 to 739, VOTER a278448771e14471ab4df1e6649485bd (127.7.39.131) evicted. New config: { opid_index: 739 OBSOLETE_local: false peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } } }
I20240501 14:10:17.873977 14816 raft_consensus.cc:2932] T a067a9d6642349c1b99effece908888c P 4015e9df09424332bc3d7a9bad781641 [term 2 FOLLOWER]: Committing config change with OpId 2.739: config changed from index 737 to 739, VOTER a278448771e14471ab4df1e6649485bd (127.7.39.131) evicted. New config: { opid_index: 739 OBSOLETE_local: false peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } } }
I20240501 14:10:17.874150 13302 catalog_manager.cc:5484] T a067a9d6642349c1b99effece908888c P 7379b25903bb465b8da2e85fd48f8a89 reported cstate change: config changed from index 737 to 739, VOTER a278448771e14471ab4df1e6649485bd (127.7.39.131) evicted. New cstate: current_term: 2 leader_uuid: "7379b25903bb465b8da2e85fd48f8a89" committed_config { opid_index: 739 OBSOLETE_local: false peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } } }
I20240501 14:10:17.876760 13687 tablet_service.cc:1514] Processing DeleteTablet for tablet a067a9d6642349c1b99effece908888c with delete_type TABLET_DATA_TOMBSTONED (TS a278448771e14471ab4df1e6649485bd not found in new config with opid_index 739) from {username='slave'} at 127.0.0.1:45074
I20240501 14:10:17.878710 18903 tablet_replica.cc:330] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd: stopping tablet replica
I20240501 14:10:17.878867 18903 raft_consensus.cc:2229] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:17.879081 18903 raft_consensus.cc:2258] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:17.880776 18903 ts_tablet_manager.cc:1875] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:17.883069 18903 ts_tablet_manager.cc:1888] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.738
I20240501 14:10:17.883160 18903 log.cc:1196] T a067a9d6642349c1b99effece908888c P a278448771e14471ab4df1e6649485bd: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal/wals/a067a9d6642349c1b99effece908888c
I20240501 14:10:17.883739 13288 catalog_manager.cc:4830] TS a278448771e14471ab4df1e6649485bd (127.7.39.131:46461): tablet a067a9d6642349c1b99effece908888c (table rebalance_test_table_4 [id=7319ca2094424c85b7008d8839665ed4]) successfully deleted
I20240501 14:10:17.934614 14665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:17.934777 15062 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:17.941875 14928 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:17.942550 13426 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:17.949550 13687 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:17.950737 13555 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:17.953624 17951 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:17.969103 14796 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:18.011710 14816 consensus_queue.cc:237] T bf0b5fd770b8405b9049c52d5d95bc53 P 4015e9df09424332bc3d7a9bad781641 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 811, Committed index: 811, Last appended: 2.811, Last appended by leader: 810, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 812 OBSOLETE_local: false peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } }
I20240501 14:10:18.012835 17971 raft_consensus.cc:1261] T bf0b5fd770b8405b9049c52d5d95bc53 P 7379b25903bb465b8da2e85fd48f8a89 [term 2 FOLLOWER]: Refusing update from remote peer 4015e9df09424332bc3d7a9bad781641: Log matching property violated. Preceding OpId in replica: term: 2 index: 811. Preceding OpId from leader: term: 2 index: 812. (index mismatch)
I20240501 14:10:18.013204 18710 consensus_queue.cc:1035] T bf0b5fd770b8405b9049c52d5d95bc53 P 4015e9df09424332bc3d7a9bad781641 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 812, Last known committed idx: 811, Time since last communication: 0.000s
I20240501 14:10:18.014608 18843 raft_consensus.cc:2932] T bf0b5fd770b8405b9049c52d5d95bc53 P 4015e9df09424332bc3d7a9bad781641 [term 2 LEADER]: Committing config change with OpId 2.812: config changed from index 810 to 812, VOTER 43736346b1ff48c78ac919c80ba58654 (127.7.39.130) evicted. New config: { opid_index: 812 OBSOLETE_local: false peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } } }
I20240501 14:10:18.015018 17971 raft_consensus.cc:2932] T bf0b5fd770b8405b9049c52d5d95bc53 P 7379b25903bb465b8da2e85fd48f8a89 [term 2 FOLLOWER]: Committing config change with OpId 2.812: config changed from index 810 to 812, VOTER 43736346b1ff48c78ac919c80ba58654 (127.7.39.130) evicted. New config: { opid_index: 812 OBSOLETE_local: false peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } } }
I20240501 14:10:18.016860 13302 catalog_manager.cc:5484] T bf0b5fd770b8405b9049c52d5d95bc53 P 4015e9df09424332bc3d7a9bad781641 reported cstate change: config changed from index 810 to 812, VOTER 43736346b1ff48c78ac919c80ba58654 (127.7.39.130) evicted. New cstate: current_term: 2 leader_uuid: "4015e9df09424332bc3d7a9bad781641" committed_config { opid_index: 812 OBSOLETE_local: false peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } } }
I20240501 14:10:18.020375 13555 tablet_service.cc:1514] Processing DeleteTablet for tablet bf0b5fd770b8405b9049c52d5d95bc53 with delete_type TABLET_DATA_TOMBSTONED (TS 43736346b1ff48c78ac919c80ba58654 not found in new config with opid_index 812) from {username='slave'} at 127.0.0.1:57610
I20240501 14:10:18.020820 18958 tablet_replica.cc:330] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654: stopping tablet replica
I20240501 14:10:18.020975 18958 raft_consensus.cc:2229] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:18.021189 18958 raft_consensus.cc:2258] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:18.022749 18958 ts_tablet_manager.cc:1875] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:18.025331 18958 ts_tablet_manager.cc:1888] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.811
I20240501 14:10:18.025413 18958 log.cc:1196] T bf0b5fd770b8405b9049c52d5d95bc53 P 43736346b1ff48c78ac919c80ba58654: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal/wals/bf0b5fd770b8405b9049c52d5d95bc53
I20240501 14:10:18.026144 13291 catalog_manager.cc:4830] TS 43736346b1ff48c78ac919c80ba58654 (127.7.39.130:43311): tablet bf0b5fd770b8405b9049c52d5d95bc53 (table rebalance_test_table_3 [id=50bc5b382cee4078bfb9cc86c185c1b2]) successfully deleted
I20240501 14:10:18.042198 14816 raft_consensus.cc:1261] T ba183686332c4ab5b606bbc5f1b4e787 P 4015e9df09424332bc3d7a9bad781641 [term 2 FOLLOWER]: Refusing update from remote peer 7379b25903bb465b8da2e85fd48f8a89: Log matching property violated. Preceding OpId in replica: term: 1 index: 853. Preceding OpId from leader: term: 2 index: 854. (index mismatch)
I20240501 14:10:18.042722 18350 consensus_queue.cc:1035] T ba183686332c4ab5b606bbc5f1b4e787 P 7379b25903bb465b8da2e85fd48f8a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 854, Last known committed idx: 853, Time since last communication: 0.000s
I20240501 14:10:18.045800 13442 raft_consensus.cc:1261] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b [term 2 FOLLOWER]: Refusing update from remote peer 7379b25903bb465b8da2e85fd48f8a89: Log matching property violated. Preceding OpId in replica: term: 1 index: 853. Preceding OpId from leader: term: 2 index: 854. (index mismatch)
I20240501 14:10:18.046216 18350 consensus_queue.cc:1035] T ba183686332c4ab5b606bbc5f1b4e787 P 7379b25903bb465b8da2e85fd48f8a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50d514f034204fc38331aa0a4fd74a1b" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 37933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 854, Last known committed idx: 853, Time since last communication: 0.000s
I20240501 14:10:18.077527 14665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:18.079200 14796 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:18.083498 13555 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:18.084744 13426 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:18.088662 13687 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:18.088662 15062 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:18.097740 17951 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:18.107663 14928 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:18.151826 13577 consensus_queue.cc:237] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } }
I20240501 14:10:18.153023 17971 raft_consensus.cc:1261] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 7379b25903bb465b8da2e85fd48f8a89 [term 3 FOLLOWER]: Refusing update from remote peer 43736346b1ff48c78ac919c80ba58654: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240501 14:10:18.153393 18559 consensus_queue.cc:1035] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:18.154892 18169 raft_consensus.cc:2932] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 3 to 5, VOTER e26df472efe547f8bc5951dce352471b (127.7.39.134) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } } }
I20240501 14:10:18.155392 17971 raft_consensus.cc:2932] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 7379b25903bb465b8da2e85fd48f8a89 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 3 to 5, VOTER e26df472efe547f8bc5951dce352471b (127.7.39.134) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } } }
I20240501 14:10:18.157407 13298 catalog_manager.cc:5484] T b0eb8ac1778a40a0a0e3c6b113ce5741 P 43736346b1ff48c78ac919c80ba58654 reported cstate change: config changed from index 3 to 5, VOTER e26df472efe547f8bc5951dce352471b (127.7.39.134) evicted. New cstate: current_term: 3 leader_uuid: "43736346b1ff48c78ac919c80ba58654" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43736346b1ff48c78ac919c80ba58654" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43311 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } } }
I20240501 14:10:18.165004 14928 tablet_service.cc:1514] Processing DeleteTablet for tablet b0eb8ac1778a40a0a0e3c6b113ce5741 with delete_type TABLET_DATA_TOMBSTONED (TS e26df472efe547f8bc5951dce352471b not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:49256
I20240501 14:10:18.165486 19017 tablet_replica.cc:330] stopping tablet replica
I20240501 14:10:18.165827 19017 raft_consensus.cc:2229] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:18.166030 19017 raft_consensus.cc:2258] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:18.167877 19017 ts_tablet_manager.cc:1875] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:18.170265 19017 ts_tablet_manager.cc:1888] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.4
I20240501 14:10:18.170360 19017 log.cc:1196] T b0eb8ac1778a40a0a0e3c6b113ce5741 P e26df472efe547f8bc5951dce352471b: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/wal/wals/b0eb8ac1778a40a0a0e3c6b113ce5741
I20240501 14:10:18.170914 13289 catalog_manager.cc:4830] TS e26df472efe547f8bc5951dce352471b (127.7.39.134:33035): tablet b0eb8ac1778a40a0a0e3c6b113ce5741 (table TestTable [id=59a0cdfa526143ab84978a90fb8bc85b]) successfully deleted
I20240501 14:10:18.216004 14665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:18.217747 13555 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:18.220345 15062 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:18.231133 13687 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:18.233273 13426 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:18.239315 17951 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:18.240707 14796 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:18.246780 14928 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:18.288900 17971 consensus_queue.cc:237] T ba183686332c4ab5b606bbc5f1b4e787 P 7379b25903bb465b8da2e85fd48f8a89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 854, Committed index: 854, Last appended: 2.854, Last appended by leader: 853, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 855 OBSOLETE_local: false peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } }
I20240501 14:10:18.290037 14816 raft_consensus.cc:1261] T ba183686332c4ab5b606bbc5f1b4e787 P 4015e9df09424332bc3d7a9bad781641 [term 2 FOLLOWER]: Refusing update from remote peer 7379b25903bb465b8da2e85fd48f8a89: Log matching property violated. Preceding OpId in replica: term: 2 index: 854. Preceding OpId from leader: term: 2 index: 855. (index mismatch)
I20240501 14:10:18.290426 18350 consensus_queue.cc:1035] T ba183686332c4ab5b606bbc5f1b4e787 P 7379b25903bb465b8da2e85fd48f8a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 855, Last known committed idx: 854, Time since last communication: 0.000s
I20240501 14:10:18.291971 18600 raft_consensus.cc:2932] T ba183686332c4ab5b606bbc5f1b4e787 P 7379b25903bb465b8da2e85fd48f8a89 [term 2 LEADER]: Committing config change with OpId 2.855: config changed from index 853 to 855, VOTER 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129) evicted. New config: { opid_index: 855 OBSOLETE_local: false peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } } }
I20240501 14:10:18.292387 14816 raft_consensus.cc:2932] T ba183686332c4ab5b606bbc5f1b4e787 P 4015e9df09424332bc3d7a9bad781641 [term 2 FOLLOWER]: Committing config change with OpId 2.855: config changed from index 853 to 855, VOTER 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129) evicted. New config: { opid_index: 855 OBSOLETE_local: false peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } } }
I20240501 14:10:18.294234 13300 catalog_manager.cc:5484] T ba183686332c4ab5b606bbc5f1b4e787 P 7379b25903bb465b8da2e85fd48f8a89 reported cstate change: config changed from index 853 to 855, VOTER 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129) evicted. New cstate: current_term: 2 leader_uuid: "7379b25903bb465b8da2e85fd48f8a89" committed_config { opid_index: 855 OBSOLETE_local: false peers { permanent_uuid: "4015e9df09424332bc3d7a9bad781641" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 42905 } } peers { permanent_uuid: "7379b25903bb465b8da2e85fd48f8a89" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 42889 } } }
I20240501 14:10:18.296751 13426 tablet_service.cc:1514] Processing DeleteTablet for tablet ba183686332c4ab5b606bbc5f1b4e787 with delete_type TABLET_DATA_TOMBSTONED (TS 50d514f034204fc38331aa0a4fd74a1b not found in new config with opid_index 855) from {username='slave'} at 127.0.0.1:45262
I20240501 14:10:18.297139 19066 tablet_replica.cc:330] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b: stopping tablet replica
I20240501 14:10:18.297260 19066 raft_consensus.cc:2229] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:18.297354 19066 raft_consensus.cc:2258] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:18.298703 19066 ts_tablet_manager.cc:1875] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:18.300684 19066 ts_tablet_manager.cc:1888] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.854
I20240501 14:10:18.300767 19066 log.cc:1196] T ba183686332c4ab5b606bbc5f1b4e787 P 50d514f034204fc38331aa0a4fd74a1b: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal/wals/ba183686332c4ab5b606bbc5f1b4e787
I20240501 14:10:18.301391 13291 catalog_manager.cc:4830] TS 50d514f034204fc38331aa0a4fd74a1b (127.7.39.129:37933): tablet ba183686332c4ab5b606bbc5f1b4e787 (table rebalance_test_table_2 [id=c8e8153727f143bdae4e267482cc7660]) successfully deleted
I20240501 14:10:18.583914 14665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:18.587669 13555 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:18.587775 13426 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:18.591606 14796 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:18.591974 13687 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:18.593464 14928 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:18.607434 17951 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:18.608067 15062 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:18.694124 14665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:18.699997 14796 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:18.701512 13555 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:18.701524 14928 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:18.707118 17951 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:18.707192 13426 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:18.707898 15062 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:18.715224 13687 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:18.828477 15062 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:18.828474 14665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:18.835201 17951 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:18.836877 13555 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:18.837996 14796 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:18.838243 13426 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:18.843684 13687 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:10:18.862468 14928 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
f86e915e5bb5479aa7f3e095cf9884ba | 127.7.39.190:41789 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.39.148:33673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
2496e4af45ba458eacaacc9d25dcf01a | 127.7.39.132:43977 | HEALTHY | <none> | 1 | 0
39bde7c058154f8aa40085db77f8c6e5 | 127.7.39.135:37645 | HEALTHY | <none> | 1 | 0
4015e9df09424332bc3d7a9bad781641 | 127.7.39.133:42905 | HEALTHY | <none> | 1 | 0
43736346b1ff48c78ac919c80ba58654 | 127.7.39.130:43311 | HEALTHY | <none> | 4 | 0
50d514f034204fc38331aa0a4fd74a1b | 127.7.39.129:37933 | HEALTHY | <none> | 2 | 0
7379b25903bb465b8da2e85fd48f8a89 | 127.7.39.136:42889 | HEALTHY | <none> | 2 | 0
a278448771e14471ab4df1e6649485bd | 127.7.39.131:46461 | HEALTHY | <none> | 3 | 0
e26df472efe547f8bc5951dce352471b | 127.7.39.134:33035 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.7.39.129 | experimental | 127.7.39.129:37933
local_ip_for_outbound_sockets | 127.7.39.130 | experimental | 127.7.39.130:43311
local_ip_for_outbound_sockets | 127.7.39.131 | experimental | 127.7.39.131:46461
local_ip_for_outbound_sockets | 127.7.39.132 | experimental | 127.7.39.132:43977
local_ip_for_outbound_sockets | 127.7.39.133 | experimental | 127.7.39.133:42905
local_ip_for_outbound_sockets | 127.7.39.134 | experimental | 127.7.39.134:33035
local_ip_for_outbound_sockets | 127.7.39.135 | experimental | 127.7.39.135:37645
local_ip_for_outbound_sockets | 127.7.39.136 | experimental | 127.7.39.136:42889
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.7.39.129:37933
server_dump_info_path | /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.39.130:43311
server_dump_info_path | /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.39.131:46461
server_dump_info_path | /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.7.39.132:43977
server_dump_info_path | /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.39.133:42905
server_dump_info_path | /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.39.134:33035
server_dump_info_path | /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.7.39.135:37645
server_dump_info_path | /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.7.39.136:42889
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.39.148:33673 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 4
Third Quartile | 4
Maximum | 4
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 14:10:18.885984 7326 log_verifier.cc:126] Checking tablet 0dc94f932e9a4319988e700aff042f22
I20240501 14:10:19.009375 7326 log_verifier.cc:177] Verified matching terms for 826 ops in tablet 0dc94f932e9a4319988e700aff042f22
I20240501 14:10:19.009624 7326 log_verifier.cc:126] Checking tablet 341c6b416c374a9ba4fc34d861ba2ef9
I20240501 14:10:19.108052 7326 log_verifier.cc:177] Verified matching terms for 680 ops in tablet 341c6b416c374a9ba4fc34d861ba2ef9
I20240501 14:10:19.108232 7326 log_verifier.cc:126] Checking tablet 3f34112bba8a43a58420148533a6e989
I20240501 14:10:19.229918 7326 log_verifier.cc:177] Verified matching terms for 845 ops in tablet 3f34112bba8a43a58420148533a6e989
I20240501 14:10:19.230160 7326 log_verifier.cc:126] Checking tablet 4d3b3213e1dd426b8efc3fdc733b1842
I20240501 14:10:19.349175 7326 log_verifier.cc:177] Verified matching terms for 789 ops in tablet 4d3b3213e1dd426b8efc3fdc733b1842
I20240501 14:10:19.349423 7326 log_verifier.cc:126] Checking tablet 8a89cc8e4b2b476ba954d3cc64537031
I20240501 14:10:19.475813 7326 log_verifier.cc:177] Verified matching terms for 829 ops in tablet 8a89cc8e4b2b476ba954d3cc64537031
I20240501 14:10:19.476032 7326 log_verifier.cc:126] Checking tablet a067a9d6642349c1b99effece908888c
I20240501 14:10:19.589008 7326 log_verifier.cc:177] Verified matching terms for 739 ops in tablet a067a9d6642349c1b99effece908888c
I20240501 14:10:19.589257 7326 log_verifier.cc:126] Checking tablet b0eb8ac1778a40a0a0e3c6b113ce5741
I20240501 14:10:19.601261 7326 log_verifier.cc:177] Verified matching terms for 5 ops in tablet b0eb8ac1778a40a0a0e3c6b113ce5741
I20240501 14:10:19.601332 7326 log_verifier.cc:126] Checking tablet ba183686332c4ab5b606bbc5f1b4e787
I20240501 14:10:19.726305 7326 log_verifier.cc:177] Verified matching terms for 855 ops in tablet ba183686332c4ab5b606bbc5f1b4e787
I20240501 14:10:19.726560 7326 log_verifier.cc:126] Checking tablet bd836361783c4b3782c0f64c6794488b
I20240501 14:10:19.836437 7326 log_verifier.cc:177] Verified matching terms for 741 ops in tablet bd836361783c4b3782c0f64c6794488b
I20240501 14:10:19.836658 7326 log_verifier.cc:126] Checking tablet bf0b5fd770b8405b9049c52d5d95bc53
I20240501 14:10:19.959544 7326 log_verifier.cc:177] Verified matching terms for 812 ops in tablet bf0b5fd770b8405b9049c52d5d95bc53
I20240501 14:10:19.959806 7326 log_verifier.cc:126] Checking tablet cce2260455ac4edbb9bc54b103c50eb3
I20240501 14:10:20.070868 7326 log_verifier.cc:177] Verified matching terms for 768 ops in tablet cce2260455ac4edbb9bc54b103c50eb3
I20240501 14:10:20.071141 7326 log_verifier.cc:126] Checking tablet cdb343c0875e4709b9637c241dd4d582
I20240501 14:10:20.185564 7326 log_verifier.cc:177] Verified matching terms for 787 ops in tablet cdb343c0875e4709b9637c241dd4d582
I20240501 14:10:20.185822 7326 log_verifier.cc:126] Checking tablet cde568b4e46a4c5499031d0e18ae51de
I20240501 14:10:20.290212 7326 log_verifier.cc:177] Verified matching terms for 730 ops in tablet cde568b4e46a4c5499031d0e18ae51de
I20240501 14:10:20.290493 7326 log_verifier.cc:126] Checking tablet d948c2d5182b415b8aedb4c7b786247f
I20240501 14:10:20.424522 7326 log_verifier.cc:177] Verified matching terms for 818 ops in tablet d948c2d5182b415b8aedb4c7b786247f
I20240501 14:10:20.424762 7326 log_verifier.cc:126] Checking tablet e4266d76760c4f84b16991f8aa38a3c3
I20240501 14:10:20.535041 7326 log_verifier.cc:177] Verified matching terms for 735 ops in tablet e4266d76760c4f84b16991f8aa38a3c3
I20240501 14:10:20.535370 7326 log_verifier.cc:126] Checking tablet f1c30f16a13444cc95132eeb9ea6c96c
I20240501 14:10:20.639035 7326 log_verifier.cc:177] Verified matching terms for 675 ops in tablet f1c30f16a13444cc95132eeb9ea6c96c
I20240501 14:10:20.640681 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 13364
I20240501 14:10:20.649572 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 13495
I20240501 14:10:20.657748 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 13626
I20240501 14:10:20.665228 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 14603
I20240501 14:10:20.671777 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 14734
I20240501 14:10:20.678050 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 14865
I20240501 14:10:20.683882 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 14999
W20240501 14:10:20.688977 17905 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.133:42905: connect: Connection refused (error 111)
W20240501 14:10:20.689862 17905 consensus_peers.cc:485] T ba183686332c4ab5b606bbc5f1b4e787 P 7379b25903bb465b8da2e85fd48f8a89 -> Peer 4015e9df09424332bc3d7a9bad781641 (127.7.39.133:42905): Couldn't send request to peer 4015e9df09424332bc3d7a9bad781641. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.133:42905: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:20.690030 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 17889
I20240501 14:10:20.695420 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 13273
2024-05-01T14:10:20Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/2 (42793 ms)
[----------] 1 test from RebalanceParamTest (42793 ms total)
[----------] 1 test from ConcurrentRebalancersTest
[ RUN ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/0
I20240501 14:10:20.752636 7326 test_util.cc:268] Using random seed: 201352950
I20240501 14:10:20.753366 7326 ts_itest-base.cc:112] Starting cluster with:
I20240501 14:10:20.753407 7326 ts_itest-base.cc:113] --------------
I20240501 14:10:20.753417 7326 ts_itest-base.cc:114] 8 tablet servers
I20240501 14:10:20.753427 7326 ts_itest-base.cc:115] 3 replicas per TS
I20240501 14:10:20.753436 7326 ts_itest-base.cc:116] --------------
2024-05-01T14:10:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T14:10:20Z Disabled control of system clock
I20240501 14:10:20.762722 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.39.190:34571
--webserver_interface=127.7.39.190
--webserver_port=0
--builtin_ntp_servers=127.7.39.148:46161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.39.190:34571
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240501 14:10:20.868256 19203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:10:20.868496 19203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:10:20.868564 19203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:10:20.871394 19203 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:10:20.871461 19203 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 14:10:20.871500 19203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:10:20.871546 19203 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 14:10:20.871585 19203 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 14:10:20.875084 19203 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:46161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.7.39.190:34571
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.39.190:34571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.7.39.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.19203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:10:20.875973 19203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:10:20.877051 19203 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:10:20.884956 19208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:10:20.886252 19209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:10:20.887272 19211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:10:20.888628 19203 server_base.cc:1034] running on GCE node
I20240501 14:10:20.889174 19203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:10:20.890103 19203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:10:20.891255 19203 hybrid_clock.cc:648] HybridClock initialized: now 1714572620891246 us; error 25 us; skew 500 ppm
I20240501 14:10:20.896672 19203 webserver.cc:460] Webserver started at http://127.7.39.190:36823/ using document root <none> and password file <none>
I20240501 14:10:20.897271 19203 fs_manager.cc:359] Metadata directory not provided
I20240501 14:10:20.897337 19203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:10:20.897536 19203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:10:20.899269 19203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "e7ca182af9f547e7b7200a15751811d3"
format_stamp: "Formatted at 2024-05-01 14:10:20 on dist-test-slave-rnfw"
I20240501 14:10:20.899741 19203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "e7ca182af9f547e7b7200a15751811d3"
format_stamp: "Formatted at 2024-05-01 14:10:20 on dist-test-slave-rnfw"
I20240501 14:10:20.902967 19203 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:10:20.905196 19217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:20.906157 19203 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:10:20.906266 19203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/wal
uuid: "e7ca182af9f547e7b7200a15751811d3"
format_stamp: "Formatted at 2024-05-01 14:10:20 on dist-test-slave-rnfw"
I20240501 14:10:20.906351 19203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:10:20.921761 19203 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:10:20.922369 19203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:10:20.922523 19203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:10:20.929023 19203 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.190:34571
I20240501 14:10:20.929033 19269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.190:34571 every 8 connection(s)
I20240501 14:10:20.929983 19203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 14:10:20.932582 19270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:20.937453 19270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7ca182af9f547e7b7200a15751811d3: Bootstrap starting.
I20240501 14:10:20.938571 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 19203
I20240501 14:10:20.938689 7326 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 14:10:20.939718 19270 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7ca182af9f547e7b7200a15751811d3: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:20.940544 19270 log.cc:824] T 00000000000000000000000000000000 P e7ca182af9f547e7b7200a15751811d3: Log is configured to *not* fsync() on all Append() calls
I20240501 14:10:20.942102 19270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7ca182af9f547e7b7200a15751811d3: No bootstrap required, opened a new log
I20240501 14:10:20.944563 19270 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7ca182af9f547e7b7200a15751811d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ca182af9f547e7b7200a15751811d3" member_type: VOTER last_known_addr { host: "127.7.39.190" port: 34571 } }
I20240501 14:10:20.944747 19270 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7ca182af9f547e7b7200a15751811d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:20.944814 19270 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7ca182af9f547e7b7200a15751811d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7ca182af9f547e7b7200a15751811d3, State: Initialized, Role: FOLLOWER
I20240501 14:10:20.945307 19270 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7ca182af9f547e7b7200a15751811d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ca182af9f547e7b7200a15751811d3" member_type: VOTER last_known_addr { host: "127.7.39.190" port: 34571 } }
I20240501 14:10:20.945458 19270 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7ca182af9f547e7b7200a15751811d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:10:20.945572 19270 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7ca182af9f547e7b7200a15751811d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:10:20.945695 19270 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7ca182af9f547e7b7200a15751811d3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:20.946589 19270 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7ca182af9f547e7b7200a15751811d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ca182af9f547e7b7200a15751811d3" member_type: VOTER last_known_addr { host: "127.7.39.190" port: 34571 } }
I20240501 14:10:20.946892 19270 leader_election.cc:302] T 00000000000000000000000000000000 P e7ca182af9f547e7b7200a15751811d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7ca182af9f547e7b7200a15751811d3; no voters:
I20240501 14:10:20.947185 19270 leader_election.cc:288] T 00000000000000000000000000000000 P e7ca182af9f547e7b7200a15751811d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:10:20.947683 19275 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7ca182af9f547e7b7200a15751811d3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:20.947908 19275 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7ca182af9f547e7b7200a15751811d3 [term 1 LEADER]: Becoming Leader. State: Replica: e7ca182af9f547e7b7200a15751811d3, State: Running, Role: LEADER
I20240501 14:10:20.948185 19275 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7ca182af9f547e7b7200a15751811d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ca182af9f547e7b7200a15751811d3" member_type: VOTER last_known_addr { host: "127.7.39.190" port: 34571 } }
I20240501 14:10:20.948464 19270 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7ca182af9f547e7b7200a15751811d3 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:10:20.949811 19276 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7ca182af9f547e7b7200a15751811d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7ca182af9f547e7b7200a15751811d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ca182af9f547e7b7200a15751811d3" member_type: VOTER last_known_addr { host: "127.7.39.190" port: 34571 } } }
I20240501 14:10:20.949934 19276 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7ca182af9f547e7b7200a15751811d3 [sys.catalog]: This master's current role is: LEADER
I20240501 14:10:20.950174 19277 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7ca182af9f547e7b7200a15751811d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7ca182af9f547e7b7200a15751811d3. Latest consensus state: current_term: 1 leader_uuid: "e7ca182af9f547e7b7200a15751811d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ca182af9f547e7b7200a15751811d3" member_type: VOTER last_known_addr { host: "127.7.39.190" port: 34571 } } }
I20240501 14:10:20.950246 19277 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7ca182af9f547e7b7200a15751811d3 [sys.catalog]: This master's current role is: LEADER
I20240501 14:10:20.950913 19283 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:10:20.953464 19283 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:10:20.957815 19283 catalog_manager.cc:1347] Generated new cluster ID: 9c1c87ee00dd4bae98d56f6a0aed20bf
I20240501 14:10:20.957877 19283 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:10:20.980444 19283 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 14:10:20.981483 19283 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:10:20.986853 19283 catalog_manager.cc:5857] T 00000000000000000000000000000000 P e7ca182af9f547e7b7200a15751811d3: Generated new TSK 0
I20240501 14:10:20.987504 19283 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:10:20.994052 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.129:0
--local_ip_for_outbound_sockets=127.7.39.129
--webserver_interface=127.7.39.129
--webserver_port=0
--tserver_master_addrs=127.7.39.190:34571
--builtin_ntp_servers=127.7.39.148:46161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 14:10:21.096122 19294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:10:21.096354 19294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:10:21.096423 19294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:10:21.099360 19294 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:10:21.099434 19294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:10:21.099517 19294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.129
I20240501 14:10:21.102819 19294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:46161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.7.39.129
--webserver_port=0
--tserver_master_addrs=127.7.39.190:34571
--never_fsync=true
--heap_profile_path=/tmp/kudu.19294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.129
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:10:21.103725 19294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:10:21.104749 19294 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:10:21.113821 19302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:10:21.114782 19299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:10:21.116168 19300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:10:21.117275 19294 server_base.cc:1034] running on GCE node
I20240501 14:10:21.117694 19294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:10:21.118237 19294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:10:21.119374 19294 hybrid_clock.cc:648] HybridClock initialized: now 1714572621119365 us; error 29 us; skew 500 ppm
I20240501 14:10:21.121399 19294 webserver.cc:460] Webserver started at http://127.7.39.129:41449/ using document root <none> and password file <none>
I20240501 14:10:21.122020 19294 fs_manager.cc:359] Metadata directory not provided
I20240501 14:10:21.122097 19294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:10:21.122304 19294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:10:21.124063 19294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc"
format_stamp: "Formatted at 2024-05-01 14:10:21 on dist-test-slave-rnfw"
I20240501 14:10:21.124583 19294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc"
format_stamp: "Formatted at 2024-05-01 14:10:21 on dist-test-slave-rnfw"
I20240501 14:10:21.128046 19294 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:10:21.130218 19308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:21.131182 19294 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:10:21.131294 19294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc"
format_stamp: "Formatted at 2024-05-01 14:10:21 on dist-test-slave-rnfw"
I20240501 14:10:21.131387 19294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:10:21.157953 19294 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:10:21.158581 19294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:10:21.158753 19294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:10:21.159302 19294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:10:21.160144 19294 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:10:21.160193 19294 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:21.160252 19294 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:10:21.160282 19294 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:21.168365 19421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.129:35847 every 8 connection(s)
I20240501 14:10:21.168346 19294 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.129:35847
I20240501 14:10:21.169404 19294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 14:10:21.170189 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 19294
I20240501 14:10:21.170305 7326 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 14:10:21.172613 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.130:0
--local_ip_for_outbound_sockets=127.7.39.130
--webserver_interface=127.7.39.130
--webserver_port=0
--tserver_master_addrs=127.7.39.190:34571
--builtin_ntp_servers=127.7.39.148:46161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:10:21.178534 19422 heartbeater.cc:347] Connected to a master server at 127.7.39.190:34571
I20240501 14:10:21.178738 19422 heartbeater.cc:464] Registering TS with master...
I20240501 14:10:21.179186 19422 heartbeater.cc:511] Master 127.7.39.190:34571 requested a full tablet report, sending...
I20240501 14:10:21.180581 19234 ts_manager.cc:194] Registered new tserver with Master: fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847)
I20240501 14:10:21.181702 19234 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.129:43639
W20240501 14:10:21.278566 19425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:10:21.278769 19425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:10:21.278812 19425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:10:21.281512 19425 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:10:21.281575 19425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:10:21.281659 19425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.130
I20240501 14:10:21.284862 19425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:46161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.7.39.130
--webserver_port=0
--tserver_master_addrs=127.7.39.190:34571
--never_fsync=true
--heap_profile_path=/tmp/kudu.19425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.130
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:10:21.285673 19425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:10:21.286733 19425 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:10:21.296031 19430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:10:21.297427 19433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:10:21.298462 19431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:10:21.299862 19425 server_base.cc:1034] running on GCE node
I20240501 14:10:21.300287 19425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:10:21.300853 19425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:10:21.302008 19425 hybrid_clock.cc:648] HybridClock initialized: now 1714572621301990 us; error 34 us; skew 500 ppm
I20240501 14:10:21.303941 19425 webserver.cc:460] Webserver started at http://127.7.39.130:41597/ using document root <none> and password file <none>
I20240501 14:10:21.304481 19425 fs_manager.cc:359] Metadata directory not provided
I20240501 14:10:21.304535 19425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:10:21.304702 19425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:10:21.306370 19425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "99a0be6ef2144b7fb247ba9d3f051d66"
format_stamp: "Formatted at 2024-05-01 14:10:21 on dist-test-slave-rnfw"
I20240501 14:10:21.306809 19425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "99a0be6ef2144b7fb247ba9d3f051d66"
format_stamp: "Formatted at 2024-05-01 14:10:21 on dist-test-slave-rnfw"
I20240501 14:10:21.310178 19425 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240501 14:10:21.312269 19439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:21.313268 19425 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 14:10:21.313400 19425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "99a0be6ef2144b7fb247ba9d3f051d66"
format_stamp: "Formatted at 2024-05-01 14:10:21 on dist-test-slave-rnfw"
I20240501 14:10:21.313482 19425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:10:21.325999 19425 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:10:21.326556 19425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:10:21.326716 19425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:10:21.327231 19425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:10:21.328053 19425 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:10:21.328102 19425 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:21.328161 19425 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:10:21.328187 19425 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:21.336328 19425 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.130:41829
I20240501 14:10:21.336364 19552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.130:41829 every 8 connection(s)
I20240501 14:10:21.337193 19425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 14:10:21.339506 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 19425
I20240501 14:10:21.339627 7326 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 14:10:21.341732 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.131:0
--local_ip_for_outbound_sockets=127.7.39.131
--webserver_interface=127.7.39.131
--webserver_port=0
--tserver_master_addrs=127.7.39.190:34571
--builtin_ntp_servers=127.7.39.148:46161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:10:21.346024 19553 heartbeater.cc:347] Connected to a master server at 127.7.39.190:34571
I20240501 14:10:21.346243 19553 heartbeater.cc:464] Registering TS with master...
I20240501 14:10:21.346680 19553 heartbeater.cc:511] Master 127.7.39.190:34571 requested a full tablet report, sending...
I20240501 14:10:21.347648 19234 ts_manager.cc:194] Registered new tserver with Master: 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829)
I20240501 14:10:21.348287 19234 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.130:33509
W20240501 14:10:21.455592 19556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:10:21.455807 19556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:10:21.455866 19556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:10:21.458710 19556 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:10:21.458768 19556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:10:21.458853 19556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.131
I20240501 14:10:21.462256 19556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:46161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.7.39.131
--webserver_port=0
--tserver_master_addrs=127.7.39.190:34571
--never_fsync=true
--heap_profile_path=/tmp/kudu.19556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.131
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:10:21.463120 19556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:10:21.464154 19556 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:10:21.472700 19564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:10:21.474001 19562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:10:21.475018 19561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:10:21.476300 19556 server_base.cc:1034] running on GCE node
I20240501 14:10:21.476688 19556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:10:21.477273 19556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:10:21.478447 19556 hybrid_clock.cc:648] HybridClock initialized: now 1714572621478435 us; error 27 us; skew 500 ppm
I20240501 14:10:21.480300 19556 webserver.cc:460] Webserver started at http://127.7.39.131:33029/ using document root <none> and password file <none>
I20240501 14:10:21.480842 19556 fs_manager.cc:359] Metadata directory not provided
I20240501 14:10:21.480895 19556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:10:21.481055 19556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:10:21.482648 19556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "ec8cde17a00645e7b38d5d29f7425367"
format_stamp: "Formatted at 2024-05-01 14:10:21 on dist-test-slave-rnfw"
I20240501 14:10:21.483130 19556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "ec8cde17a00645e7b38d5d29f7425367"
format_stamp: "Formatted at 2024-05-01 14:10:21 on dist-test-slave-rnfw"
I20240501 14:10:21.486593 19556 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:10:21.488849 19570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:21.489840 19556 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240501 14:10:21.489945 19556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "ec8cde17a00645e7b38d5d29f7425367"
format_stamp: "Formatted at 2024-05-01 14:10:21 on dist-test-slave-rnfw"
I20240501 14:10:21.490031 19556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:10:21.525476 19556 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:10:21.526126 19556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:10:21.526293 19556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:10:21.526818 19556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:10:21.527690 19556 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:10:21.527740 19556 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:21.527796 19556 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:10:21.527825 19556 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:21.535691 19556 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.131:45201
I20240501 14:10:21.535710 19683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.131:45201 every 8 connection(s)
I20240501 14:10:21.536626 19556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 14:10:21.538233 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 19556
I20240501 14:10:21.538362 7326 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 14:10:21.540566 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.132:0
--local_ip_for_outbound_sockets=127.7.39.132
--webserver_interface=127.7.39.132
--webserver_port=0
--tserver_master_addrs=127.7.39.190:34571
--builtin_ntp_servers=127.7.39.148:46161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:10:21.545238 19684 heartbeater.cc:347] Connected to a master server at 127.7.39.190:34571
I20240501 14:10:21.545464 19684 heartbeater.cc:464] Registering TS with master...
I20240501 14:10:21.545873 19684 heartbeater.cc:511] Master 127.7.39.190:34571 requested a full tablet report, sending...
I20240501 14:10:21.546788 19234 ts_manager.cc:194] Registered new tserver with Master: ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201)
I20240501 14:10:21.547343 19234 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.131:56325
W20240501 14:10:21.644145 19687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:10:21.644369 19687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:10:21.644409 19687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:10:21.647146 19687 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:10:21.647207 19687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:10:21.647291 19687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.132
I20240501 14:10:21.650671 19687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:46161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.7.39.132
--webserver_port=0
--tserver_master_addrs=127.7.39.190:34571
--never_fsync=true
--heap_profile_path=/tmp/kudu.19687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.132
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:10:21.651566 19687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:10:21.652653 19687 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:10:21.661798 19693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:10:21.662680 19695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:10:21.664113 19692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:10:21.665181 19687 server_base.cc:1034] running on GCE node
I20240501 14:10:21.665575 19687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:10:21.666132 19687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:10:21.667294 19687 hybrid_clock.cc:648] HybridClock initialized: now 1714572621667280 us; error 28 us; skew 500 ppm
I20240501 14:10:21.675254 19687 webserver.cc:460] Webserver started at http://127.7.39.132:42631/ using document root <none> and password file <none>
I20240501 14:10:21.675902 19687 fs_manager.cc:359] Metadata directory not provided
I20240501 14:10:21.676014 19687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:10:21.676193 19687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:10:21.677790 19687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "da3c4e948ff747678f18456c134d7ed4"
format_stamp: "Formatted at 2024-05-01 14:10:21 on dist-test-slave-rnfw"
I20240501 14:10:21.678279 19687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "da3c4e948ff747678f18456c134d7ed4"
format_stamp: "Formatted at 2024-05-01 14:10:21 on dist-test-slave-rnfw"
I20240501 14:10:21.681593 19687 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 14:10:21.683738 19701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:21.684679 19687 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:10:21.684782 19687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "da3c4e948ff747678f18456c134d7ed4"
format_stamp: "Formatted at 2024-05-01 14:10:21 on dist-test-slave-rnfw"
I20240501 14:10:21.684860 19687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:10:21.696471 19687 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:10:21.697005 19687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:10:21.697163 19687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:10:21.697651 19687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:10:21.698477 19687 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:10:21.698526 19687 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:21.698580 19687 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:10:21.698611 19687 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:21.707366 19687 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.132:35357
I20240501 14:10:21.707401 19814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.132:35357 every 8 connection(s)
I20240501 14:10:21.708248 19687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 14:10:21.716732 19815 heartbeater.cc:347] Connected to a master server at 127.7.39.190:34571
I20240501 14:10:21.716969 19815 heartbeater.cc:464] Registering TS with master...
I20240501 14:10:21.717187 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 19687
I20240501 14:10:21.717308 7326 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 14:10:21.717514 19815 heartbeater.cc:511] Master 127.7.39.190:34571 requested a full tablet report, sending...
I20240501 14:10:21.718487 19234 ts_manager.cc:194] Registered new tserver with Master: da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357)
I20240501 14:10:21.719228 19234 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.132:43201
I20240501 14:10:21.719864 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.133:0
--local_ip_for_outbound_sockets=127.7.39.133
--webserver_interface=127.7.39.133
--webserver_port=0
--tserver_master_addrs=127.7.39.190:34571
--builtin_ntp_servers=127.7.39.148:46161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 14:10:21.828567 19818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:10:21.828790 19818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:10:21.828835 19818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:10:21.831931 19818 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:10:21.831997 19818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:10:21.832067 19818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.133
I20240501 14:10:21.835678 19818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:46161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.7.39.133
--webserver_port=0
--tserver_master_addrs=127.7.39.190:34571
--never_fsync=true
--heap_profile_path=/tmp/kudu.19818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.133
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:10:21.836546 19818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:10:21.837615 19818 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:10:21.846714 19826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:10:21.847674 19823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:10:21.849102 19824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:10:21.850198 19818 server_base.cc:1034] running on GCE node
I20240501 14:10:21.850579 19818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:10:21.851209 19818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:10:21.852387 19818 hybrid_clock.cc:648] HybridClock initialized: now 1714572621852368 us; error 39 us; skew 500 ppm
I20240501 14:10:21.854456 19818 webserver.cc:460] Webserver started at http://127.7.39.133:33707/ using document root <none> and password file <none>
I20240501 14:10:21.855011 19818 fs_manager.cc:359] Metadata directory not provided
I20240501 14:10:21.855101 19818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:10:21.855291 19818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:10:21.856871 19818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "5bf81f3d986e428190d3f7727f36d96c"
format_stamp: "Formatted at 2024-05-01 14:10:21 on dist-test-slave-rnfw"
I20240501 14:10:21.857316 19818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "5bf81f3d986e428190d3f7727f36d96c"
format_stamp: "Formatted at 2024-05-01 14:10:21 on dist-test-slave-rnfw"
I20240501 14:10:21.860533 19818 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 14:10:21.862563 19832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:21.863534 19818 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:10:21.863627 19818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "5bf81f3d986e428190d3f7727f36d96c"
format_stamp: "Formatted at 2024-05-01 14:10:21 on dist-test-slave-rnfw"
I20240501 14:10:21.863708 19818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:10:21.891978 19818 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:10:21.892619 19818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:10:21.892779 19818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:10:21.893326 19818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:10:21.894302 19818 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:10:21.894352 19818 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:21.894399 19818 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:10:21.894429 19818 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:21.902650 19818 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.133:46883
I20240501 14:10:21.902683 19945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.133:46883 every 8 connection(s)
I20240501 14:10:21.903645 19818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 14:10:21.906628 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 19818
I20240501 14:10:21.906754 7326 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 14:10:21.909031 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.134:0
--local_ip_for_outbound_sockets=127.7.39.134
--webserver_interface=127.7.39.134
--webserver_port=0
--tserver_master_addrs=127.7.39.190:34571
--builtin_ntp_servers=127.7.39.148:46161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:10:21.912983 19946 heartbeater.cc:347] Connected to a master server at 127.7.39.190:34571
I20240501 14:10:21.913259 19946 heartbeater.cc:464] Registering TS with master...
I20240501 14:10:21.913870 19946 heartbeater.cc:511] Master 127.7.39.190:34571 requested a full tablet report, sending...
I20240501 14:10:21.914969 19234 ts_manager.cc:194] Registered new tserver with Master: 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133:46883)
I20240501 14:10:21.915794 19234 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.133:59175
W20240501 14:10:22.016485 19949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:10:22.016706 19949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:10:22.016750 19949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:10:22.019639 19949 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:10:22.019697 19949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:10:22.019762 19949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.134
I20240501 14:10:22.023104 19949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:46161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.7.39.134
--webserver_port=0
--tserver_master_addrs=127.7.39.190:34571
--never_fsync=true
--heap_profile_path=/tmp/kudu.19949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.134
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:10:22.023913 19949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:10:22.024993 19949 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:10:22.033798 19957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:10:22.034723 19955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:10:22.036070 19954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:10:22.037202 19949 server_base.cc:1034] running on GCE node
I20240501 14:10:22.037606 19949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:10:22.038138 19949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:10:22.039304 19949 hybrid_clock.cc:648] HybridClock initialized: now 1714572622039268 us; error 37 us; skew 500 ppm
I20240501 14:10:22.041184 19949 webserver.cc:460] Webserver started at http://127.7.39.134:42995/ using document root <none> and password file <none>
I20240501 14:10:22.041714 19949 fs_manager.cc:359] Metadata directory not provided
I20240501 14:10:22.041774 19949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:10:22.041934 19949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:10:22.043589 19949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "e6246b183293412494919f1e7efc9448"
format_stamp: "Formatted at 2024-05-01 14:10:22 on dist-test-slave-rnfw"
I20240501 14:10:22.044024 19949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "e6246b183293412494919f1e7efc9448"
format_stamp: "Formatted at 2024-05-01 14:10:22 on dist-test-slave-rnfw"
I20240501 14:10:22.047102 19949 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:10:22.049177 19963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:22.050165 19949 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:10:22.050261 19949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "e6246b183293412494919f1e7efc9448"
format_stamp: "Formatted at 2024-05-01 14:10:22 on dist-test-slave-rnfw"
I20240501 14:10:22.050349 19949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:10:22.059887 19949 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:10:22.060405 19949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:10:22.060554 19949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:10:22.061051 19949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:10:22.061822 19949 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:10:22.061874 19949 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:22.061929 19949 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:10:22.061954 19949 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:22.070194 19949 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.134:41911
I20240501 14:10:22.070227 20076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.134:41911 every 8 connection(s)
I20240501 14:10:22.071028 19949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 14:10:22.075634 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 19949
I20240501 14:10:22.075759 7326 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 14:10:22.078246 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.135:0
--local_ip_for_outbound_sockets=127.7.39.135
--webserver_interface=127.7.39.135
--webserver_port=0
--tserver_master_addrs=127.7.39.190:34571
--builtin_ntp_servers=127.7.39.148:46161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:10:22.079746 20077 heartbeater.cc:347] Connected to a master server at 127.7.39.190:34571
I20240501 14:10:22.079998 20077 heartbeater.cc:464] Registering TS with master...
I20240501 14:10:22.080451 20077 heartbeater.cc:511] Master 127.7.39.190:34571 requested a full tablet report, sending...
I20240501 14:10:22.081372 19234 ts_manager.cc:194] Registered new tserver with Master: e6246b183293412494919f1e7efc9448 (127.7.39.134:41911)
I20240501 14:10:22.082177 19234 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.134:60645
W20240501 14:10:22.183422 20080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:10:22.183624 20080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:10:22.183678 20080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 14:10:22.183862 19422 heartbeater.cc:503] Master 127.7.39.190:34571 was elected leader, sending a full tablet report...
W20240501 14:10:22.186709 20080 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:10:22.186781 20080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:10:22.186879 20080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.135
I20240501 14:10:22.190150 20080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:46161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.7.39.135
--webserver_port=0
--tserver_master_addrs=127.7.39.190:34571
--never_fsync=true
--heap_profile_path=/tmp/kudu.20080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.135
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:10:22.191017 20080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:10:22.192054 20080 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:10:22.200664 20085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:10:22.201912 20088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:10:22.203033 20086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:10:22.204209 20080 server_base.cc:1034] running on GCE node
I20240501 14:10:22.204576 20080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:10:22.205113 20080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:10:22.206272 20080 hybrid_clock.cc:648] HybridClock initialized: now 1714572622206247 us; error 32 us; skew 500 ppm
I20240501 14:10:22.208086 20080 webserver.cc:460] Webserver started at http://127.7.39.135:33933/ using document root <none> and password file <none>
I20240501 14:10:22.208632 20080 fs_manager.cc:359] Metadata directory not provided
I20240501 14:10:22.208693 20080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:10:22.208861 20080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:10:22.210533 20080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "43ce4a4c954f4a5f80954b38933c189d"
format_stamp: "Formatted at 2024-05-01 14:10:22 on dist-test-slave-rnfw"
I20240501 14:10:22.211006 20080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "43ce4a4c954f4a5f80954b38933c189d"
format_stamp: "Formatted at 2024-05-01 14:10:22 on dist-test-slave-rnfw"
I20240501 14:10:22.214206 20080 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20240501 14:10:22.216219 20094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:22.217209 20080 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 14:10:22.217319 20080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "43ce4a4c954f4a5f80954b38933c189d"
format_stamp: "Formatted at 2024-05-01 14:10:22 on dist-test-slave-rnfw"
I20240501 14:10:22.217401 20080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:10:22.233574 20080 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:10:22.234130 20080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:10:22.234303 20080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:10:22.234818 20080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:10:22.235739 20080 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:10:22.235798 20080 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:22.235863 20080 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:10:22.235893 20080 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:22.243898 20080 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.135:38751
I20240501 14:10:22.243928 20207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.135:38751 every 8 connection(s)
I20240501 14:10:22.244731 20080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 14:10:22.245121 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 20080
I20240501 14:10:22.245244 7326 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 14:10:22.247859 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.136:0
--local_ip_for_outbound_sockets=127.7.39.136
--webserver_interface=127.7.39.136
--webserver_port=0
--tserver_master_addrs=127.7.39.190:34571
--builtin_ntp_servers=127.7.39.148:46161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:10:22.254369 20208 heartbeater.cc:347] Connected to a master server at 127.7.39.190:34571
I20240501 14:10:22.254591 20208 heartbeater.cc:464] Registering TS with master...
I20240501 14:10:22.255034 20208 heartbeater.cc:511] Master 127.7.39.190:34571 requested a full tablet report, sending...
I20240501 14:10:22.256079 19234 ts_manager.cc:194] Registered new tserver with Master: 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135:38751)
I20240501 14:10:22.256851 19234 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.135:38903
I20240501 14:10:22.350639 19553 heartbeater.cc:503] Master 127.7.39.190:34571 was elected leader, sending a full tablet report...
W20240501 14:10:22.354616 20211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:10:22.354856 20211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:10:22.354916 20211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:10:22.357820 20211 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:10:22.357878 20211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:10:22.357942 20211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.136
I20240501 14:10:22.361122 20211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:46161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.7.39.136
--webserver_port=0
--tserver_master_addrs=127.7.39.190:34571
--never_fsync=true
--heap_profile_path=/tmp/kudu.20211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.136
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:10:22.361881 20211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:10:22.362932 20211 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:10:22.372033 20217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:10:22.372917 20216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:10:22.374323 20219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:10:22.375396 20211 server_base.cc:1034] running on GCE node
I20240501 14:10:22.375767 20211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:10:22.376263 20211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:10:22.377411 20211 hybrid_clock.cc:648] HybridClock initialized: now 1714572622377401 us; error 27 us; skew 500 ppm
I20240501 14:10:22.379369 20211 webserver.cc:460] Webserver started at http://127.7.39.136:42151/ using document root <none> and password file <none>
I20240501 14:10:22.379892 20211 fs_manager.cc:359] Metadata directory not provided
I20240501 14:10:22.379948 20211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:10:22.380137 20211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:10:22.381724 20211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "778c78105092428bbcde75ab2cdb6a05"
format_stamp: "Formatted at 2024-05-01 14:10:22 on dist-test-slave-rnfw"
I20240501 14:10:22.382234 20211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "778c78105092428bbcde75ab2cdb6a05"
format_stamp: "Formatted at 2024-05-01 14:10:22 on dist-test-slave-rnfw"
I20240501 14:10:22.385586 20211 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20240501 14:10:22.387696 20225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:22.388661 20211 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:10:22.388767 20211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "778c78105092428bbcde75ab2cdb6a05"
format_stamp: "Formatted at 2024-05-01 14:10:22 on dist-test-slave-rnfw"
I20240501 14:10:22.388857 20211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:10:22.403326 20211 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:10:22.403889 20211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:10:22.404047 20211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:10:22.404567 20211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:10:22.405443 20211 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:10:22.405495 20211 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:22.405555 20211 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:10:22.405586 20211 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:22.414779 20211 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.136:34895
I20240501 14:10:22.414853 20338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.136:34895 every 8 connection(s)
I20240501 14:10:22.415886 20211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 14:10:22.423766 20339 heartbeater.cc:347] Connected to a master server at 127.7.39.190:34571
I20240501 14:10:22.423990 20339 heartbeater.cc:464] Registering TS with master...
I20240501 14:10:22.424463 20339 heartbeater.cc:511] Master 127.7.39.190:34571 requested a full tablet report, sending...
I20240501 14:10:22.424494 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 20211
I20240501 14:10:22.424602 7326 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 14:10:22.425487 19234 ts_manager.cc:194] Registered new tserver with Master: 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136:34895)
I20240501 14:10:22.426133 19234 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.136:35381
I20240501 14:10:22.426836 7326 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 14:10:22.435647 7326 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 14:10:22.435715 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 19818
I20240501 14:10:22.441435 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 19949
I20240501 14:10:22.446527 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 20080
I20240501 14:10:22.451835 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 20211
I20240501 14:10:22.549254 19684 heartbeater.cc:503] Master 127.7.39.190:34571 was elected leader, sending a full tablet report...
I20240501 14:10:22.721424 19815 heartbeater.cc:503] Master 127.7.39.190:34571 was elected leader, sending a full tablet report...
I20240501 14:10:26.208392 19233 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45968:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:10:26.229084 19487 tablet_service.cc:1467] Processing CreateTablet for tablet 1d408b075d784953a57554bcdf19158a (DEFAULT_TABLE table=rebalance_test_table_0 [id=b52269ef7c494f74a5e4f0d5cb3b8a47]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:26.229641 19749 tablet_service.cc:1467] Processing CreateTablet for tablet 1d408b075d784953a57554bcdf19158a (DEFAULT_TABLE table=rebalance_test_table_0 [id=b52269ef7c494f74a5e4f0d5cb3b8a47]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:26.230033 19487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d408b075d784953a57554bcdf19158a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:26.230417 19749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d408b075d784953a57554bcdf19158a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:26.231318 19748 tablet_service.cc:1467] Processing CreateTablet for tablet 85593473500b4d5eaf9f8b0bf2296777 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b52269ef7c494f74a5e4f0d5cb3b8a47]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:26.231577 19748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85593473500b4d5eaf9f8b0bf2296777. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:26.233582 19617 tablet_service.cc:1467] Processing CreateTablet for tablet 85593473500b4d5eaf9f8b0bf2296777 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b52269ef7c494f74a5e4f0d5cb3b8a47]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:26.233584 19618 tablet_service.cc:1467] Processing CreateTablet for tablet 55830424a98348e382759c54afdeabcd (DEFAULT_TABLE table=rebalance_test_table_0 [id=b52269ef7c494f74a5e4f0d5cb3b8a47]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:26.233997 19356 tablet_service.cc:1467] Processing CreateTablet for tablet 1d408b075d784953a57554bcdf19158a (DEFAULT_TABLE table=rebalance_test_table_0 [id=b52269ef7c494f74a5e4f0d5cb3b8a47]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:26.234288 19354 tablet_service.cc:1467] Processing CreateTablet for tablet 85593473500b4d5eaf9f8b0bf2296777 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b52269ef7c494f74a5e4f0d5cb3b8a47]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:26.234503 19618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55830424a98348e382759c54afdeabcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:26.234527 19355 tablet_service.cc:1467] Processing CreateTablet for tablet 55830424a98348e382759c54afdeabcd (DEFAULT_TABLE table=rebalance_test_table_0 [id=b52269ef7c494f74a5e4f0d5cb3b8a47]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:26.234779 19356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d408b075d784953a57554bcdf19158a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:26.235370 19354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85593473500b4d5eaf9f8b0bf2296777. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:26.236841 19355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55830424a98348e382759c54afdeabcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:26.238369 20359 tablet_bootstrap.cc:492] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4: Bootstrap starting.
I20240501 14:10:26.240021 19486 tablet_service.cc:1467] Processing CreateTablet for tablet 55830424a98348e382759c54afdeabcd (DEFAULT_TABLE table=rebalance_test_table_0 [id=b52269ef7c494f74a5e4f0d5cb3b8a47]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:26.240341 19486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55830424a98348e382759c54afdeabcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:26.240578 20359 tablet_bootstrap.cc:654] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:26.241369 20359 log.cc:824] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4: Log is configured to *not* fsync() on all Append() calls
I20240501 14:10:26.242349 20361 tablet_bootstrap.cc:492] T 85593473500b4d5eaf9f8b0bf2296777 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Bootstrap starting.
I20240501 14:10:26.243522 19617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85593473500b4d5eaf9f8b0bf2296777. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:26.244331 20361 tablet_bootstrap.cc:654] T 85593473500b4d5eaf9f8b0bf2296777 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:26.245644 20361 log.cc:824] T 85593473500b4d5eaf9f8b0bf2296777 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Log is configured to *not* fsync() on all Append() calls
I20240501 14:10:26.246614 20363 tablet_bootstrap.cc:492] T 55830424a98348e382759c54afdeabcd P 99a0be6ef2144b7fb247ba9d3f051d66: Bootstrap starting.
I20240501 14:10:26.247336 20360 tablet_bootstrap.cc:492] T 55830424a98348e382759c54afdeabcd P ec8cde17a00645e7b38d5d29f7425367: Bootstrap starting.
I20240501 14:10:26.247877 20361 tablet_bootstrap.cc:492] T 85593473500b4d5eaf9f8b0bf2296777 P fb1e6bc2d07b4ce3b622a66f14c93bdc: No bootstrap required, opened a new log
I20240501 14:10:26.248113 20361 ts_tablet_manager.cc:1367] T 85593473500b4d5eaf9f8b0bf2296777 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20240501 14:10:26.248489 20359 tablet_bootstrap.cc:492] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4: No bootstrap required, opened a new log
I20240501 14:10:26.248565 20363 tablet_bootstrap.cc:654] T 55830424a98348e382759c54afdeabcd P 99a0be6ef2144b7fb247ba9d3f051d66: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:26.248692 20359 ts_tablet_manager.cc:1367] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20240501 14:10:26.249411 20363 log.cc:824] T 55830424a98348e382759c54afdeabcd P 99a0be6ef2144b7fb247ba9d3f051d66: Log is configured to *not* fsync() on all Append() calls
I20240501 14:10:26.250627 20360 tablet_bootstrap.cc:654] T 55830424a98348e382759c54afdeabcd P ec8cde17a00645e7b38d5d29f7425367: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:26.251011 20363 tablet_bootstrap.cc:492] T 55830424a98348e382759c54afdeabcd P 99a0be6ef2144b7fb247ba9d3f051d66: No bootstrap required, opened a new log
I20240501 14:10:26.251204 20363 ts_tablet_manager.cc:1367] T 55830424a98348e382759c54afdeabcd P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 14:10:26.251511 20360 log.cc:824] T 55830424a98348e382759c54afdeabcd P ec8cde17a00645e7b38d5d29f7425367: Log is configured to *not* fsync() on all Append() calls
I20240501 14:10:26.251575 20361 raft_consensus.cc:348] T 85593473500b4d5eaf9f8b0bf2296777 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:26.251792 20361 raft_consensus.cc:374] T 85593473500b4d5eaf9f8b0bf2296777 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:26.251849 20361 raft_consensus.cc:729] T 85593473500b4d5eaf9f8b0bf2296777 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1e6bc2d07b4ce3b622a66f14c93bdc, State: Initialized, Role: FOLLOWER
I20240501 14:10:26.252315 20361 consensus_queue.cc:260] T 85593473500b4d5eaf9f8b0bf2296777 P fb1e6bc2d07b4ce3b622a66f14c93bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:26.252385 20359 raft_consensus.cc:348] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:26.252517 20359 raft_consensus.cc:374] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:26.252545 20359 raft_consensus.cc:729] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3c4e948ff747678f18456c134d7ed4, State: Initialized, Role: FOLLOWER
I20240501 14:10:26.252948 20359 consensus_queue.cc:260] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:26.253593 20359 ts_tablet_manager.cc:1398] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240501 14:10:26.253334 20361 ts_tablet_manager.cc:1398] T 85593473500b4d5eaf9f8b0bf2296777 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent starting tablet: real 0.005s user 0.003s sys 0.001s
I20240501 14:10:26.253908 20361 tablet_bootstrap.cc:492] T 1d408b075d784953a57554bcdf19158a P fb1e6bc2d07b4ce3b622a66f14c93bdc: Bootstrap starting.
I20240501 14:10:26.253882 20359 tablet_bootstrap.cc:492] T 85593473500b4d5eaf9f8b0bf2296777 P da3c4e948ff747678f18456c134d7ed4: Bootstrap starting.
I20240501 14:10:26.254676 20363 raft_consensus.cc:348] T 55830424a98348e382759c54afdeabcd P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:26.254833 20359 tablet_bootstrap.cc:654] T 85593473500b4d5eaf9f8b0bf2296777 P da3c4e948ff747678f18456c134d7ed4: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:26.254849 20361 tablet_bootstrap.cc:654] T 1d408b075d784953a57554bcdf19158a P fb1e6bc2d07b4ce3b622a66f14c93bdc: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:26.254844 20363 raft_consensus.cc:374] T 55830424a98348e382759c54afdeabcd P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:26.254890 20363 raft_consensus.cc:729] T 55830424a98348e382759c54afdeabcd P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a0be6ef2144b7fb247ba9d3f051d66, State: Initialized, Role: FOLLOWER
I20240501 14:10:26.255908 20361 tablet_bootstrap.cc:492] T 1d408b075d784953a57554bcdf19158a P fb1e6bc2d07b4ce3b622a66f14c93bdc: No bootstrap required, opened a new log
I20240501 14:10:26.255986 20361 ts_tablet_manager.cc:1367] T 1d408b075d784953a57554bcdf19158a P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:10:26.256496 20361 raft_consensus.cc:348] T 1d408b075d784953a57554bcdf19158a P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:26.256610 20361 raft_consensus.cc:374] T 1d408b075d784953a57554bcdf19158a P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:26.256644 20361 raft_consensus.cc:729] T 1d408b075d784953a57554bcdf19158a P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1e6bc2d07b4ce3b622a66f14c93bdc, State: Initialized, Role: FOLLOWER
I20240501 14:10:26.256773 20361 consensus_queue.cc:260] T 1d408b075d784953a57554bcdf19158a P fb1e6bc2d07b4ce3b622a66f14c93bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:26.257000 20359 tablet_bootstrap.cc:492] T 85593473500b4d5eaf9f8b0bf2296777 P da3c4e948ff747678f18456c134d7ed4: No bootstrap required, opened a new log
I20240501 14:10:26.257050 20361 ts_tablet_manager.cc:1398] T 1d408b075d784953a57554bcdf19158a P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:26.257073 20359 ts_tablet_manager.cc:1367] T 85593473500b4d5eaf9f8b0bf2296777 P da3c4e948ff747678f18456c134d7ed4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:10:26.257205 20361 tablet_bootstrap.cc:492] T 55830424a98348e382759c54afdeabcd P fb1e6bc2d07b4ce3b622a66f14c93bdc: Bootstrap starting.
I20240501 14:10:26.257522 20359 raft_consensus.cc:348] T 85593473500b4d5eaf9f8b0bf2296777 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:26.257612 20359 raft_consensus.cc:374] T 85593473500b4d5eaf9f8b0bf2296777 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:26.257627 20360 tablet_bootstrap.cc:492] T 55830424a98348e382759c54afdeabcd P ec8cde17a00645e7b38d5d29f7425367: No bootstrap required, opened a new log
I20240501 14:10:26.257643 20359 raft_consensus.cc:729] T 85593473500b4d5eaf9f8b0bf2296777 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3c4e948ff747678f18456c134d7ed4, State: Initialized, Role: FOLLOWER
I20240501 14:10:26.257773 20360 ts_tablet_manager.cc:1367] T 55830424a98348e382759c54afdeabcd P ec8cde17a00645e7b38d5d29f7425367: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240501 14:10:26.257755 20359 consensus_queue.cc:260] T 85593473500b4d5eaf9f8b0bf2296777 P da3c4e948ff747678f18456c134d7ed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:26.257992 20361 tablet_bootstrap.cc:654] T 55830424a98348e382759c54afdeabcd P fb1e6bc2d07b4ce3b622a66f14c93bdc: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:26.257995 20359 ts_tablet_manager.cc:1398] T 85593473500b4d5eaf9f8b0bf2296777 P da3c4e948ff747678f18456c134d7ed4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:26.259032 20363 consensus_queue.cc:260] T 55830424a98348e382759c54afdeabcd P 99a0be6ef2144b7fb247ba9d3f051d66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:26.259994 20363 ts_tablet_manager.cc:1398] T 55830424a98348e382759c54afdeabcd P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent starting tablet: real 0.009s user 0.002s sys 0.003s
I20240501 14:10:26.260739 20363 tablet_bootstrap.cc:492] T 1d408b075d784953a57554bcdf19158a P 99a0be6ef2144b7fb247ba9d3f051d66: Bootstrap starting.
I20240501 14:10:26.260761 20361 tablet_bootstrap.cc:492] T 55830424a98348e382759c54afdeabcd P fb1e6bc2d07b4ce3b622a66f14c93bdc: No bootstrap required, opened a new log
I20240501 14:10:26.260830 20361 ts_tablet_manager.cc:1367] T 55830424a98348e382759c54afdeabcd P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:10:26.261222 20361 raft_consensus.cc:348] T 55830424a98348e382759c54afdeabcd P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:26.261324 20361 raft_consensus.cc:374] T 55830424a98348e382759c54afdeabcd P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:26.261355 20361 raft_consensus.cc:729] T 55830424a98348e382759c54afdeabcd P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1e6bc2d07b4ce3b622a66f14c93bdc, State: Initialized, Role: FOLLOWER
I20240501 14:10:26.261515 20361 consensus_queue.cc:260] T 55830424a98348e382759c54afdeabcd P fb1e6bc2d07b4ce3b622a66f14c93bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:26.261574 20363 tablet_bootstrap.cc:654] T 1d408b075d784953a57554bcdf19158a P 99a0be6ef2144b7fb247ba9d3f051d66: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:26.261864 20361 ts_tablet_manager.cc:1398] T 55830424a98348e382759c54afdeabcd P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 14:10:26.262693 20360 raft_consensus.cc:348] T 55830424a98348e382759c54afdeabcd P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:26.262857 20360 raft_consensus.cc:374] T 55830424a98348e382759c54afdeabcd P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:26.262903 20360 raft_consensus.cc:729] T 55830424a98348e382759c54afdeabcd P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec8cde17a00645e7b38d5d29f7425367, State: Initialized, Role: FOLLOWER
I20240501 14:10:26.263340 20363 tablet_bootstrap.cc:492] T 1d408b075d784953a57554bcdf19158a P 99a0be6ef2144b7fb247ba9d3f051d66: No bootstrap required, opened a new log
I20240501 14:10:26.263347 20360 consensus_queue.cc:260] T 55830424a98348e382759c54afdeabcd P ec8cde17a00645e7b38d5d29f7425367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:26.263450 20363 ts_tablet_manager.cc:1367] T 1d408b075d784953a57554bcdf19158a P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:10:26.264168 20360 ts_tablet_manager.cc:1398] T 55830424a98348e382759c54afdeabcd P ec8cde17a00645e7b38d5d29f7425367: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240501 14:10:26.264442 20360 tablet_bootstrap.cc:492] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367: Bootstrap starting.
I20240501 14:10:26.264997 20363 raft_consensus.cc:348] T 1d408b075d784953a57554bcdf19158a P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:26.265126 20363 raft_consensus.cc:374] T 1d408b075d784953a57554bcdf19158a P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:26.265180 20363 raft_consensus.cc:729] T 1d408b075d784953a57554bcdf19158a P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a0be6ef2144b7fb247ba9d3f051d66, State: Initialized, Role: FOLLOWER
I20240501 14:10:26.265316 20363 consensus_queue.cc:260] T 1d408b075d784953a57554bcdf19158a P 99a0be6ef2144b7fb247ba9d3f051d66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:26.265395 20360 tablet_bootstrap.cc:654] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:26.265647 20363 ts_tablet_manager.cc:1398] T 1d408b075d784953a57554bcdf19158a P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:10:26.266450 20360 tablet_bootstrap.cc:492] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367: No bootstrap required, opened a new log
I20240501 14:10:26.266541 20360 ts_tablet_manager.cc:1367] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:10:26.266975 20360 raft_consensus.cc:348] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:26.267100 20360 raft_consensus.cc:374] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:26.267145 20360 raft_consensus.cc:729] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec8cde17a00645e7b38d5d29f7425367, State: Initialized, Role: FOLLOWER
I20240501 14:10:26.267324 20360 consensus_queue.cc:260] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:26.267674 20360 ts_tablet_manager.cc:1398] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20240501 14:10:26.290807 19685 tablet.cc:2369] T 55830424a98348e382759c54afdeabcd P ec8cde17a00645e7b38d5d29f7425367: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:10:26.303900 20374 raft_consensus.cc:482] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:26.304092 20374 raft_consensus.cc:504] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:26.305361 20374 leader_election.cc:288] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357), fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847)
I20240501 14:10:26.309043 19769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85593473500b4d5eaf9f8b0bf2296777" candidate_uuid: "ec8cde17a00645e7b38d5d29f7425367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3c4e948ff747678f18456c134d7ed4" is_pre_election: true
I20240501 14:10:26.309183 19376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85593473500b4d5eaf9f8b0bf2296777" candidate_uuid: "ec8cde17a00645e7b38d5d29f7425367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" is_pre_election: true
I20240501 14:10:26.309350 19769 raft_consensus.cc:2454] T 85593473500b4d5eaf9f8b0bf2296777 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec8cde17a00645e7b38d5d29f7425367 in term 0.
I20240501 14:10:26.309480 19376 raft_consensus.cc:2454] T 85593473500b4d5eaf9f8b0bf2296777 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec8cde17a00645e7b38d5d29f7425367 in term 0.
I20240501 14:10:26.309773 19573 leader_election.cc:302] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da3c4e948ff747678f18456c134d7ed4, ec8cde17a00645e7b38d5d29f7425367; no voters:
I20240501 14:10:26.310078 20374 raft_consensus.cc:2781] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:26.310158 20374 raft_consensus.cc:482] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:26.310214 20374 raft_consensus.cc:3037] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:26.311174 20374 raft_consensus.cc:504] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:26.311498 20374 leader_election.cc:288] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 [CANDIDATE]: Term 1 election: Requested vote from peers da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357), fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847)
I20240501 14:10:26.311843 19376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85593473500b4d5eaf9f8b0bf2296777" candidate_uuid: "ec8cde17a00645e7b38d5d29f7425367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc"
I20240501 14:10:26.311842 19769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85593473500b4d5eaf9f8b0bf2296777" candidate_uuid: "ec8cde17a00645e7b38d5d29f7425367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3c4e948ff747678f18456c134d7ed4"
I20240501 14:10:26.311954 19376 raft_consensus.cc:3037] T 85593473500b4d5eaf9f8b0bf2296777 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:26.311986 19769 raft_consensus.cc:3037] T 85593473500b4d5eaf9f8b0bf2296777 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:26.312817 19769 raft_consensus.cc:2454] T 85593473500b4d5eaf9f8b0bf2296777 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec8cde17a00645e7b38d5d29f7425367 in term 1.
I20240501 14:10:26.313087 19573 leader_election.cc:302] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da3c4e948ff747678f18456c134d7ed4, ec8cde17a00645e7b38d5d29f7425367; no voters:
I20240501 14:10:26.313253 20374 raft_consensus.cc:2781] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:26.313266 19376 raft_consensus.cc:2454] T 85593473500b4d5eaf9f8b0bf2296777 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec8cde17a00645e7b38d5d29f7425367 in term 1.
I20240501 14:10:26.313498 20374 raft_consensus.cc:686] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Becoming Leader. State: Replica: ec8cde17a00645e7b38d5d29f7425367, State: Running, Role: LEADER
I20240501 14:10:26.313863 20374 consensus_queue.cc:237] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:26.316540 19233 catalog_manager.cc:5484] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131). New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } }
I20240501 14:10:26.336683 20372 raft_consensus.cc:482] T 55830424a98348e382759c54afdeabcd P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:26.336922 20372 raft_consensus.cc:504] T 55830424a98348e382759c54afdeabcd P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:26.338014 20372 leader_election.cc:288] T 55830424a98348e382759c54afdeabcd P 99a0be6ef2144b7fb247ba9d3f051d66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201), fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847)
W20240501 14:10:26.341728 19554 tablet.cc:2369] T 1d408b075d784953a57554bcdf19158a P 99a0be6ef2144b7fb247ba9d3f051d66: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:10:26.342435 19638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55830424a98348e382759c54afdeabcd" candidate_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec8cde17a00645e7b38d5d29f7425367" is_pre_election: true
I20240501 14:10:26.342643 19638 raft_consensus.cc:2454] T 55830424a98348e382759c54afdeabcd P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99a0be6ef2144b7fb247ba9d3f051d66 in term 0.
I20240501 14:10:26.342957 19376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55830424a98348e382759c54afdeabcd" candidate_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" is_pre_election: true
I20240501 14:10:26.343123 19376 raft_consensus.cc:2454] T 55830424a98348e382759c54afdeabcd P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99a0be6ef2144b7fb247ba9d3f051d66 in term 0.
I20240501 14:10:26.343108 19441 leader_election.cc:302] T 55830424a98348e382759c54afdeabcd P 99a0be6ef2144b7fb247ba9d3f051d66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99a0be6ef2144b7fb247ba9d3f051d66, ec8cde17a00645e7b38d5d29f7425367; no voters:
I20240501 14:10:26.343351 20372 raft_consensus.cc:2781] T 55830424a98348e382759c54afdeabcd P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:26.343423 20372 raft_consensus.cc:482] T 55830424a98348e382759c54afdeabcd P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:26.343480 20372 raft_consensus.cc:3037] T 55830424a98348e382759c54afdeabcd P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:26.344331 20372 raft_consensus.cc:504] T 55830424a98348e382759c54afdeabcd P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:26.344671 20372 leader_election.cc:288] T 55830424a98348e382759c54afdeabcd P 99a0be6ef2144b7fb247ba9d3f051d66 [CANDIDATE]: Term 1 election: Requested vote from peers ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201), fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847)
I20240501 14:10:26.345069 19376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55830424a98348e382759c54afdeabcd" candidate_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc"
I20240501 14:10:26.345093 19638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55830424a98348e382759c54afdeabcd" candidate_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec8cde17a00645e7b38d5d29f7425367"
I20240501 14:10:26.345178 19376 raft_consensus.cc:3037] T 55830424a98348e382759c54afdeabcd P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:26.345217 19638 raft_consensus.cc:3037] T 55830424a98348e382759c54afdeabcd P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:26.345919 19376 raft_consensus.cc:2454] T 55830424a98348e382759c54afdeabcd P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99a0be6ef2144b7fb247ba9d3f051d66 in term 1.
I20240501 14:10:26.346189 19440 leader_election.cc:302] T 55830424a98348e382759c54afdeabcd P 99a0be6ef2144b7fb247ba9d3f051d66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99a0be6ef2144b7fb247ba9d3f051d66, fb1e6bc2d07b4ce3b622a66f14c93bdc; no voters:
I20240501 14:10:26.346341 20372 raft_consensus.cc:2781] T 55830424a98348e382759c54afdeabcd P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:26.346462 19638 raft_consensus.cc:2454] T 55830424a98348e382759c54afdeabcd P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99a0be6ef2144b7fb247ba9d3f051d66 in term 1.
I20240501 14:10:26.346607 20372 raft_consensus.cc:686] T 55830424a98348e382759c54afdeabcd P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 LEADER]: Becoming Leader. State: Replica: 99a0be6ef2144b7fb247ba9d3f051d66, State: Running, Role: LEADER
I20240501 14:10:26.346958 20372 consensus_queue.cc:237] T 55830424a98348e382759c54afdeabcd P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:26.349473 19233 catalog_manager.cc:5484] T 55830424a98348e382759c54afdeabcd P 99a0be6ef2144b7fb247ba9d3f051d66 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130). New cstate: current_term: 1 leader_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } }
I20240501 14:10:26.398577 20368 raft_consensus.cc:482] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:26.398770 20368 raft_consensus.cc:504] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:26.399919 20368 leader_election.cc:288] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829), fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847)
I20240501 14:10:26.404422 19376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d408b075d784953a57554bcdf19158a" candidate_uuid: "da3c4e948ff747678f18456c134d7ed4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" is_pre_election: true
I20240501 14:10:26.404568 19376 raft_consensus.cc:2454] T 1d408b075d784953a57554bcdf19158a P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da3c4e948ff747678f18456c134d7ed4 in term 0.
I20240501 14:10:26.404634 19507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d408b075d784953a57554bcdf19158a" candidate_uuid: "da3c4e948ff747678f18456c134d7ed4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" is_pre_election: true
I20240501 14:10:26.404894 19507 raft_consensus.cc:2454] T 1d408b075d784953a57554bcdf19158a P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da3c4e948ff747678f18456c134d7ed4 in term 0.
I20240501 14:10:26.404937 19702 leader_election.cc:302] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da3c4e948ff747678f18456c134d7ed4, fb1e6bc2d07b4ce3b622a66f14c93bdc; no voters:
I20240501 14:10:26.405201 20368 raft_consensus.cc:2781] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:26.405273 20368 raft_consensus.cc:482] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:26.405315 20368 raft_consensus.cc:3037] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:26.406427 20368 raft_consensus.cc:504] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:26.406764 20368 leader_election.cc:288] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4 [CANDIDATE]: Term 1 election: Requested vote from peers 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829), fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847)
I20240501 14:10:26.407066 19507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d408b075d784953a57554bcdf19158a" candidate_uuid: "da3c4e948ff747678f18456c134d7ed4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a0be6ef2144b7fb247ba9d3f051d66"
I20240501 14:10:26.407119 19376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d408b075d784953a57554bcdf19158a" candidate_uuid: "da3c4e948ff747678f18456c134d7ed4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc"
I20240501 14:10:26.407172 19507 raft_consensus.cc:3037] T 1d408b075d784953a57554bcdf19158a P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:26.407230 19376 raft_consensus.cc:3037] T 1d408b075d784953a57554bcdf19158a P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:26.407953 19507 raft_consensus.cc:2454] T 1d408b075d784953a57554bcdf19158a P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da3c4e948ff747678f18456c134d7ed4 in term 1.
I20240501 14:10:26.407967 19376 raft_consensus.cc:2454] T 1d408b075d784953a57554bcdf19158a P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da3c4e948ff747678f18456c134d7ed4 in term 1.
I20240501 14:10:26.408246 19702 leader_election.cc:302] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da3c4e948ff747678f18456c134d7ed4, fb1e6bc2d07b4ce3b622a66f14c93bdc; no voters:
I20240501 14:10:26.408402 20368 raft_consensus.cc:2781] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:26.408617 20368 raft_consensus.cc:686] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4 [term 1 LEADER]: Becoming Leader. State: Replica: da3c4e948ff747678f18456c134d7ed4, State: Running, Role: LEADER
I20240501 14:10:26.408944 20368 consensus_queue.cc:237] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:26.411603 19233 catalog_manager.cc:5484] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4 reported cstate change: term changed from 0 to 1, leader changed from <none> to da3c4e948ff747678f18456c134d7ed4 (127.7.39.132). New cstate: current_term: 1 leader_uuid: "da3c4e948ff747678f18456c134d7ed4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } }
W20240501 14:10:26.424150 19423 tablet.cc:2369] T 1d408b075d784953a57554bcdf19158a P fb1e6bc2d07b4ce3b622a66f14c93bdc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:10:26.462607 19816 tablet.cc:2369] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:10:26.530318 20388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:10:26.530553 20388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:10:26.533360 20388 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240501 14:10:26.612043 19376 raft_consensus.cc:1261] T 1d408b075d784953a57554bcdf19158a P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:26.612126 19507 raft_consensus.cc:1261] T 1d408b075d784953a57554bcdf19158a P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:26.612680 20368 consensus_queue.cc:1035] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:26.612874 20386 consensus_queue.cc:1035] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:26.614492 19375 raft_consensus.cc:1261] T 85593473500b4d5eaf9f8b0bf2296777 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:26.614996 20374 consensus_queue.cc:1035] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:26.615588 19769 raft_consensus.cc:1261] T 85593473500b4d5eaf9f8b0bf2296777 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:26.617120 20374 consensus_queue.cc:1035] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:26.620841 19374 raft_consensus.cc:1261] T 55830424a98348e382759c54afdeabcd P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Refusing update from remote peer 99a0be6ef2144b7fb247ba9d3f051d66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:26.621098 19638 raft_consensus.cc:1261] T 55830424a98348e382759c54afdeabcd P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Refusing update from remote peer 99a0be6ef2144b7fb247ba9d3f051d66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:26.621655 20372 consensus_queue.cc:1035] T 55830424a98348e382759c54afdeabcd P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:26.623901 20372 consensus_queue.cc:1035] T 55830424a98348e382759c54afdeabcd P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:26.628147 20428 mvcc.cc:205] Tried to move back new op lower bound from 7022889478606958592 to 7022889477384089600. Current Snapshot: MvccSnapshot[applied={T|T < 7022889478606958592}]
I20240501 14:10:26.629693 20430 mvcc.cc:205] Tried to move back new op lower bound from 7022889478595571712 to 7022889477773541376. Current Snapshot: MvccSnapshot[applied={T|T < 7022889478595571712}]
I20240501 14:10:26.629899 20423 mvcc.cc:205] Tried to move back new op lower bound from 7022889478595571712 to 7022889477773541376. Current Snapshot: MvccSnapshot[applied={T|T < 7022889478595571712}]
I20240501 14:10:26.638355 20422 mvcc.cc:205] Tried to move back new op lower bound from 7022889478624727040 to 7022889477384089600. Current Snapshot: MvccSnapshot[applied={T|T < 7022889478606958592}]
INSERT report
rows total: 2000
time total: 111.16 ms
time per row: 0.05558 ms
I20240501 14:10:26.683164 19234 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45968:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:10:26.688251 19487 tablet_service.cc:1467] Processing CreateTablet for tablet dab5e5722e2e4b0eaea73288d8dce265 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a9b1c37e923e49a7b30905624977117c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:26.688233 19617 tablet_service.cc:1467] Processing CreateTablet for tablet dab5e5722e2e4b0eaea73288d8dce265 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a9b1c37e923e49a7b30905624977117c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:26.688552 19617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dab5e5722e2e4b0eaea73288d8dce265. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:26.689019 19618 tablet_service.cc:1467] Processing CreateTablet for tablet 661aaf83e3474966a257fcc039ef4784 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a9b1c37e923e49a7b30905624977117c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:26.689244 19618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 661aaf83e3474966a257fcc039ef4784. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:26.689213 19355 tablet_service.cc:1467] Processing CreateTablet for tablet dadab886b03540579d138cdb5b25e286 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a9b1c37e923e49a7b30905624977117c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:26.689455 19355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dadab886b03540579d138cdb5b25e286. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:26.690869 20360 tablet_bootstrap.cc:492] T dab5e5722e2e4b0eaea73288d8dce265 P ec8cde17a00645e7b38d5d29f7425367: Bootstrap starting.
I20240501 14:10:26.690897 19486 tablet_service.cc:1467] Processing CreateTablet for tablet 661aaf83e3474966a257fcc039ef4784 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a9b1c37e923e49a7b30905624977117c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:26.691658 20361 tablet_bootstrap.cc:492] T dadab886b03540579d138cdb5b25e286 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Bootstrap starting.
I20240501 14:10:26.691658 19618 tablet_service.cc:1467] Processing CreateTablet for tablet dadab886b03540579d138cdb5b25e286 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a9b1c37e923e49a7b30905624977117c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:26.691812 19485 tablet_service.cc:1467] Processing CreateTablet for tablet dadab886b03540579d138cdb5b25e286 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a9b1c37e923e49a7b30905624977117c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:26.691876 19618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dadab886b03540579d138cdb5b25e286. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:26.691919 20360 tablet_bootstrap.cc:654] T dab5e5722e2e4b0eaea73288d8dce265 P ec8cde17a00645e7b38d5d29f7425367: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:26.692500 20361 tablet_bootstrap.cc:654] T dadab886b03540579d138cdb5b25e286 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:26.692754 19748 tablet_service.cc:1467] Processing CreateTablet for tablet dab5e5722e2e4b0eaea73288d8dce265 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a9b1c37e923e49a7b30905624977117c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:26.693011 20360 tablet_bootstrap.cc:492] T dab5e5722e2e4b0eaea73288d8dce265 P ec8cde17a00645e7b38d5d29f7425367: No bootstrap required, opened a new log
I20240501 14:10:26.693022 19748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dab5e5722e2e4b0eaea73288d8dce265. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:26.693086 20360 ts_tablet_manager.cc:1367] T dab5e5722e2e4b0eaea73288d8dce265 P ec8cde17a00645e7b38d5d29f7425367: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:10:26.688551 19487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dab5e5722e2e4b0eaea73288d8dce265. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:26.693468 19486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 661aaf83e3474966a257fcc039ef4784. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:26.693503 20360 raft_consensus.cc:348] T dab5e5722e2e4b0eaea73288d8dce265 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:26.693604 20360 raft_consensus.cc:374] T dab5e5722e2e4b0eaea73288d8dce265 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:26.693636 20360 raft_consensus.cc:729] T dab5e5722e2e4b0eaea73288d8dce265 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec8cde17a00645e7b38d5d29f7425367, State: Initialized, Role: FOLLOWER
I20240501 14:10:26.693753 20360 consensus_queue.cc:260] T dab5e5722e2e4b0eaea73288d8dce265 P ec8cde17a00645e7b38d5d29f7425367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:26.694002 20360 ts_tablet_manager.cc:1398] T dab5e5722e2e4b0eaea73288d8dce265 P ec8cde17a00645e7b38d5d29f7425367: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:26.694150 20360 tablet_bootstrap.cc:492] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367: Bootstrap starting.
I20240501 14:10:26.695266 20360 tablet_bootstrap.cc:654] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:26.695233 19749 tablet_service.cc:1467] Processing CreateTablet for tablet 661aaf83e3474966a257fcc039ef4784 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a9b1c37e923e49a7b30905624977117c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:26.695431 19749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 661aaf83e3474966a257fcc039ef4784. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:26.695549 20359 tablet_bootstrap.cc:492] T dab5e5722e2e4b0eaea73288d8dce265 P da3c4e948ff747678f18456c134d7ed4: Bootstrap starting.
I20240501 14:10:26.696291 20359 tablet_bootstrap.cc:654] T dab5e5722e2e4b0eaea73288d8dce265 P da3c4e948ff747678f18456c134d7ed4: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:26.696403 19485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dadab886b03540579d138cdb5b25e286. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:26.696542 20360 tablet_bootstrap.cc:492] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367: No bootstrap required, opened a new log
I20240501 14:10:26.696607 20360 ts_tablet_manager.cc:1367] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:10:26.696857 20363 tablet_bootstrap.cc:492] T 661aaf83e3474966a257fcc039ef4784 P 99a0be6ef2144b7fb247ba9d3f051d66: Bootstrap starting.
I20240501 14:10:26.697000 20360 raft_consensus.cc:348] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:26.697088 20360 raft_consensus.cc:374] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:26.697125 20360 raft_consensus.cc:729] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec8cde17a00645e7b38d5d29f7425367, State: Initialized, Role: FOLLOWER
I20240501 14:10:26.697228 20360 consensus_queue.cc:260] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:26.697652 20360 ts_tablet_manager.cc:1398] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:26.697682 20363 tablet_bootstrap.cc:654] T 661aaf83e3474966a257fcc039ef4784 P 99a0be6ef2144b7fb247ba9d3f051d66: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:26.698578 20360 tablet_bootstrap.cc:492] T dadab886b03540579d138cdb5b25e286 P ec8cde17a00645e7b38d5d29f7425367: Bootstrap starting.
I20240501 14:10:26.698949 20363 tablet_bootstrap.cc:492] T 661aaf83e3474966a257fcc039ef4784 P 99a0be6ef2144b7fb247ba9d3f051d66: No bootstrap required, opened a new log
I20240501 14:10:26.699071 20361 tablet_bootstrap.cc:492] T dadab886b03540579d138cdb5b25e286 P fb1e6bc2d07b4ce3b622a66f14c93bdc: No bootstrap required, opened a new log
I20240501 14:10:26.699085 20363 ts_tablet_manager.cc:1367] T 661aaf83e3474966a257fcc039ef4784 P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:10:26.699148 20361 ts_tablet_manager.cc:1367] T dadab886b03540579d138cdb5b25e286 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20240501 14:10:26.699621 20361 raft_consensus.cc:348] T dadab886b03540579d138cdb5b25e286 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:26.699720 20361 raft_consensus.cc:374] T dadab886b03540579d138cdb5b25e286 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:26.699649 20363 raft_consensus.cc:348] T 661aaf83e3474966a257fcc039ef4784 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:26.699752 20361 raft_consensus.cc:729] T dadab886b03540579d138cdb5b25e286 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1e6bc2d07b4ce3b622a66f14c93bdc, State: Initialized, Role: FOLLOWER
I20240501 14:10:26.699769 20363 raft_consensus.cc:374] T 661aaf83e3474966a257fcc039ef4784 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:26.699811 20363 raft_consensus.cc:729] T 661aaf83e3474966a257fcc039ef4784 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a0be6ef2144b7fb247ba9d3f051d66, State: Initialized, Role: FOLLOWER
I20240501 14:10:26.699885 20361 consensus_queue.cc:260] T dadab886b03540579d138cdb5b25e286 P fb1e6bc2d07b4ce3b622a66f14c93bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:26.699926 20363 consensus_queue.cc:260] T 661aaf83e3474966a257fcc039ef4784 P 99a0be6ef2144b7fb247ba9d3f051d66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:26.700016 20360 tablet_bootstrap.cc:654] T dadab886b03540579d138cdb5b25e286 P ec8cde17a00645e7b38d5d29f7425367: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:26.700083 20359 tablet_bootstrap.cc:492] T dab5e5722e2e4b0eaea73288d8dce265 P da3c4e948ff747678f18456c134d7ed4: No bootstrap required, opened a new log
I20240501 14:10:26.700152 20359 ts_tablet_manager.cc:1367] T dab5e5722e2e4b0eaea73288d8dce265 P da3c4e948ff747678f18456c134d7ed4: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:10:26.700179 20363 ts_tablet_manager.cc:1398] T 661aaf83e3474966a257fcc039ef4784 P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:26.700201 20361 ts_tablet_manager.cc:1398] T dadab886b03540579d138cdb5b25e286 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:10:26.700342 20363 tablet_bootstrap.cc:492] T dab5e5722e2e4b0eaea73288d8dce265 P 99a0be6ef2144b7fb247ba9d3f051d66: Bootstrap starting.
I20240501 14:10:26.700845 20359 raft_consensus.cc:348] T dab5e5722e2e4b0eaea73288d8dce265 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:26.700933 20359 raft_consensus.cc:374] T dab5e5722e2e4b0eaea73288d8dce265 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:26.700964 20359 raft_consensus.cc:729] T dab5e5722e2e4b0eaea73288d8dce265 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3c4e948ff747678f18456c134d7ed4, State: Initialized, Role: FOLLOWER
I20240501 14:10:26.701073 20359 consensus_queue.cc:260] T dab5e5722e2e4b0eaea73288d8dce265 P da3c4e948ff747678f18456c134d7ed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:26.701161 20363 tablet_bootstrap.cc:654] T dab5e5722e2e4b0eaea73288d8dce265 P 99a0be6ef2144b7fb247ba9d3f051d66: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:26.701388 20359 ts_tablet_manager.cc:1398] T dab5e5722e2e4b0eaea73288d8dce265 P da3c4e948ff747678f18456c134d7ed4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:26.701547 20359 tablet_bootstrap.cc:492] T 661aaf83e3474966a257fcc039ef4784 P da3c4e948ff747678f18456c134d7ed4: Bootstrap starting.
I20240501 14:10:26.702368 20359 tablet_bootstrap.cc:654] T 661aaf83e3474966a257fcc039ef4784 P da3c4e948ff747678f18456c134d7ed4: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:26.703173 20363 tablet_bootstrap.cc:492] T dab5e5722e2e4b0eaea73288d8dce265 P 99a0be6ef2144b7fb247ba9d3f051d66: No bootstrap required, opened a new log
I20240501 14:10:26.703239 20363 ts_tablet_manager.cc:1367] T dab5e5722e2e4b0eaea73288d8dce265 P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:10:26.703688 20360 tablet_bootstrap.cc:492] T dadab886b03540579d138cdb5b25e286 P ec8cde17a00645e7b38d5d29f7425367: No bootstrap required, opened a new log
I20240501 14:10:26.703647 20363 raft_consensus.cc:348] T dab5e5722e2e4b0eaea73288d8dce265 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:26.703759 20363 raft_consensus.cc:374] T dab5e5722e2e4b0eaea73288d8dce265 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:26.703797 20363 raft_consensus.cc:729] T dab5e5722e2e4b0eaea73288d8dce265 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a0be6ef2144b7fb247ba9d3f051d66, State: Initialized, Role: FOLLOWER
I20240501 14:10:26.703892 20360 ts_tablet_manager.cc:1367] T dadab886b03540579d138cdb5b25e286 P ec8cde17a00645e7b38d5d29f7425367: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:10:26.703923 20363 consensus_queue.cc:260] T dab5e5722e2e4b0eaea73288d8dce265 P 99a0be6ef2144b7fb247ba9d3f051d66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:26.704187 20363 ts_tablet_manager.cc:1398] T dab5e5722e2e4b0eaea73288d8dce265 P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:10:26.704344 20363 tablet_bootstrap.cc:492] T dadab886b03540579d138cdb5b25e286 P 99a0be6ef2144b7fb247ba9d3f051d66: Bootstrap starting.
I20240501 14:10:26.704316 20360 raft_consensus.cc:348] T dadab886b03540579d138cdb5b25e286 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:26.704408 20360 raft_consensus.cc:374] T dadab886b03540579d138cdb5b25e286 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:26.704442 20360 raft_consensus.cc:729] T dadab886b03540579d138cdb5b25e286 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec8cde17a00645e7b38d5d29f7425367, State: Initialized, Role: FOLLOWER
I20240501 14:10:26.704569 20360 consensus_queue.cc:260] T dadab886b03540579d138cdb5b25e286 P ec8cde17a00645e7b38d5d29f7425367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:26.704879 20360 ts_tablet_manager.cc:1398] T dadab886b03540579d138cdb5b25e286 P ec8cde17a00645e7b38d5d29f7425367: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:26.705121 20359 tablet_bootstrap.cc:492] T 661aaf83e3474966a257fcc039ef4784 P da3c4e948ff747678f18456c134d7ed4: No bootstrap required, opened a new log
I20240501 14:10:26.705157 20363 tablet_bootstrap.cc:654] T dadab886b03540579d138cdb5b25e286 P 99a0be6ef2144b7fb247ba9d3f051d66: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:26.705195 20359 ts_tablet_manager.cc:1367] T 661aaf83e3474966a257fcc039ef4784 P da3c4e948ff747678f18456c134d7ed4: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:10:26.705662 20359 raft_consensus.cc:348] T 661aaf83e3474966a257fcc039ef4784 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:26.705756 20359 raft_consensus.cc:374] T 661aaf83e3474966a257fcc039ef4784 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:26.705796 20359 raft_consensus.cc:729] T 661aaf83e3474966a257fcc039ef4784 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3c4e948ff747678f18456c134d7ed4, State: Initialized, Role: FOLLOWER
I20240501 14:10:26.705919 20359 consensus_queue.cc:260] T 661aaf83e3474966a257fcc039ef4784 P da3c4e948ff747678f18456c134d7ed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:26.706153 20359 ts_tablet_manager.cc:1398] T 661aaf83e3474966a257fcc039ef4784 P da3c4e948ff747678f18456c134d7ed4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:26.706919 20363 tablet_bootstrap.cc:492] T dadab886b03540579d138cdb5b25e286 P 99a0be6ef2144b7fb247ba9d3f051d66: No bootstrap required, opened a new log
I20240501 14:10:26.706992 20363 ts_tablet_manager.cc:1367] T dadab886b03540579d138cdb5b25e286 P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:10:26.707429 20363 raft_consensus.cc:348] T dadab886b03540579d138cdb5b25e286 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:26.707542 20363 raft_consensus.cc:374] T dadab886b03540579d138cdb5b25e286 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:26.707635 20363 raft_consensus.cc:729] T dadab886b03540579d138cdb5b25e286 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a0be6ef2144b7fb247ba9d3f051d66, State: Initialized, Role: FOLLOWER
I20240501 14:10:26.707794 20363 consensus_queue.cc:260] T dadab886b03540579d138cdb5b25e286 P 99a0be6ef2144b7fb247ba9d3f051d66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:26.708110 20363 ts_tablet_manager.cc:1398] T dadab886b03540579d138cdb5b25e286 P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:26.720738 20374 raft_consensus.cc:482] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:26.720856 20374 raft_consensus.cc:504] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:26.721292 20374 leader_election.cc:288] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357), 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829)
I20240501 14:10:26.721680 19769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "661aaf83e3474966a257fcc039ef4784" candidate_uuid: "ec8cde17a00645e7b38d5d29f7425367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3c4e948ff747678f18456c134d7ed4" is_pre_election: true
I20240501 14:10:26.721830 19769 raft_consensus.cc:2454] T 661aaf83e3474966a257fcc039ef4784 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec8cde17a00645e7b38d5d29f7425367 in term 0.
I20240501 14:10:26.722134 19573 leader_election.cc:302] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da3c4e948ff747678f18456c134d7ed4, ec8cde17a00645e7b38d5d29f7425367; no voters:
I20240501 14:10:26.722311 20374 raft_consensus.cc:2781] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:26.722383 20374 raft_consensus.cc:482] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:26.722420 20374 raft_consensus.cc:3037] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:26.723510 20374 raft_consensus.cc:504] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:26.723858 20374 leader_election.cc:288] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 [CANDIDATE]: Term 1 election: Requested vote from peers da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357), 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829)
I20240501 14:10:26.724193 19769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "661aaf83e3474966a257fcc039ef4784" candidate_uuid: "ec8cde17a00645e7b38d5d29f7425367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3c4e948ff747678f18456c134d7ed4"
I20240501 14:10:26.724344 19769 raft_consensus.cc:3037] T 661aaf83e3474966a257fcc039ef4784 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:26.725142 19769 raft_consensus.cc:2454] T 661aaf83e3474966a257fcc039ef4784 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec8cde17a00645e7b38d5d29f7425367 in term 1.
I20240501 14:10:26.725311 19507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "661aaf83e3474966a257fcc039ef4784" candidate_uuid: "ec8cde17a00645e7b38d5d29f7425367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" is_pre_election: true
I20240501 14:10:26.725384 19506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "661aaf83e3474966a257fcc039ef4784" candidate_uuid: "ec8cde17a00645e7b38d5d29f7425367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a0be6ef2144b7fb247ba9d3f051d66"
I20240501 14:10:26.725445 19507 raft_consensus.cc:2454] T 661aaf83e3474966a257fcc039ef4784 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec8cde17a00645e7b38d5d29f7425367 in term 0.
I20240501 14:10:26.725446 19573 leader_election.cc:302] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da3c4e948ff747678f18456c134d7ed4, ec8cde17a00645e7b38d5d29f7425367; no voters:
I20240501 14:10:26.725746 20374 raft_consensus.cc:2781] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:26.725848 20374 raft_consensus.cc:686] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Becoming Leader. State: Replica: ec8cde17a00645e7b38d5d29f7425367, State: Running, Role: LEADER
I20240501 14:10:26.725982 20374 consensus_queue.cc:237] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:26.727401 19234 catalog_manager.cc:5484] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131). New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } }
I20240501 14:10:26.868031 20386 raft_consensus.cc:482] T dab5e5722e2e4b0eaea73288d8dce265 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:26.868176 20386 raft_consensus.cc:504] T dab5e5722e2e4b0eaea73288d8dce265 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:26.868624 20386 leader_election.cc:288] T dab5e5722e2e4b0eaea73288d8dce265 P da3c4e948ff747678f18456c134d7ed4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201), 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829)
I20240501 14:10:26.869436 19506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dab5e5722e2e4b0eaea73288d8dce265" candidate_uuid: "da3c4e948ff747678f18456c134d7ed4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" is_pre_election: true
I20240501 14:10:26.869583 19506 raft_consensus.cc:2454] T dab5e5722e2e4b0eaea73288d8dce265 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da3c4e948ff747678f18456c134d7ed4 in term 0.
I20240501 14:10:26.869884 19703 leader_election.cc:302] T dab5e5722e2e4b0eaea73288d8dce265 P da3c4e948ff747678f18456c134d7ed4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99a0be6ef2144b7fb247ba9d3f051d66, da3c4e948ff747678f18456c134d7ed4; no voters:
I20240501 14:10:26.870080 20386 raft_consensus.cc:2781] T dab5e5722e2e4b0eaea73288d8dce265 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:26.870124 20386 raft_consensus.cc:482] T dab5e5722e2e4b0eaea73288d8dce265 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:26.870177 20386 raft_consensus.cc:3037] T dab5e5722e2e4b0eaea73288d8dce265 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:26.871114 20386 raft_consensus.cc:504] T dab5e5722e2e4b0eaea73288d8dce265 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:26.871440 20386 leader_election.cc:288] T dab5e5722e2e4b0eaea73288d8dce265 P da3c4e948ff747678f18456c134d7ed4 [CANDIDATE]: Term 1 election: Requested vote from peers ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201), 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829)
I20240501 14:10:26.871845 19506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dab5e5722e2e4b0eaea73288d8dce265" candidate_uuid: "da3c4e948ff747678f18456c134d7ed4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a0be6ef2144b7fb247ba9d3f051d66"
I20240501 14:10:26.871964 19506 raft_consensus.cc:3037] T dab5e5722e2e4b0eaea73288d8dce265 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:26.872206 19638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dab5e5722e2e4b0eaea73288d8dce265" candidate_uuid: "da3c4e948ff747678f18456c134d7ed4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec8cde17a00645e7b38d5d29f7425367" is_pre_election: true
I20240501 14:10:26.872298 19637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dab5e5722e2e4b0eaea73288d8dce265" candidate_uuid: "da3c4e948ff747678f18456c134d7ed4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec8cde17a00645e7b38d5d29f7425367"
I20240501 14:10:26.872365 19638 raft_consensus.cc:2454] T dab5e5722e2e4b0eaea73288d8dce265 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da3c4e948ff747678f18456c134d7ed4 in term 0.
I20240501 14:10:26.872900 19506 raft_consensus.cc:2454] T dab5e5722e2e4b0eaea73288d8dce265 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da3c4e948ff747678f18456c134d7ed4 in term 1.
I20240501 14:10:26.873163 19703 leader_election.cc:302] T dab5e5722e2e4b0eaea73288d8dce265 P da3c4e948ff747678f18456c134d7ed4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99a0be6ef2144b7fb247ba9d3f051d66, da3c4e948ff747678f18456c134d7ed4; no voters: ec8cde17a00645e7b38d5d29f7425367
I20240501 14:10:26.873320 20386 raft_consensus.cc:2781] T dab5e5722e2e4b0eaea73288d8dce265 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:26.873409 20386 raft_consensus.cc:686] T dab5e5722e2e4b0eaea73288d8dce265 P da3c4e948ff747678f18456c134d7ed4 [term 1 LEADER]: Becoming Leader. State: Replica: da3c4e948ff747678f18456c134d7ed4, State: Running, Role: LEADER
I20240501 14:10:26.873567 20386 consensus_queue.cc:237] T dab5e5722e2e4b0eaea73288d8dce265 P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:26.875306 19234 catalog_manager.cc:5484] T dab5e5722e2e4b0eaea73288d8dce265 P da3c4e948ff747678f18456c134d7ed4 reported cstate change: term changed from 0 to 1, leader changed from <none> to da3c4e948ff747678f18456c134d7ed4 (127.7.39.132). New cstate: current_term: 1 leader_uuid: "da3c4e948ff747678f18456c134d7ed4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } }
I20240501 14:10:27.006810 20374 raft_consensus.cc:482] T dadab886b03540579d138cdb5b25e286 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:27.007014 20374 raft_consensus.cc:504] T dadab886b03540579d138cdb5b25e286 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:27.007563 20374 leader_election.cc:288] T dadab886b03540579d138cdb5b25e286 P ec8cde17a00645e7b38d5d29f7425367 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829), fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847)
I20240501 14:10:27.008052 19375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dadab886b03540579d138cdb5b25e286" candidate_uuid: "ec8cde17a00645e7b38d5d29f7425367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" is_pre_election: true
I20240501 14:10:27.008054 19506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dadab886b03540579d138cdb5b25e286" candidate_uuid: "ec8cde17a00645e7b38d5d29f7425367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" is_pre_election: true
I20240501 14:10:27.008200 19375 raft_consensus.cc:2454] T dadab886b03540579d138cdb5b25e286 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec8cde17a00645e7b38d5d29f7425367 in term 0.
I20240501 14:10:27.008291 19506 raft_consensus.cc:2454] T dadab886b03540579d138cdb5b25e286 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec8cde17a00645e7b38d5d29f7425367 in term 0.
I20240501 14:10:27.008514 19571 leader_election.cc:302] T dadab886b03540579d138cdb5b25e286 P ec8cde17a00645e7b38d5d29f7425367 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ec8cde17a00645e7b38d5d29f7425367, fb1e6bc2d07b4ce3b622a66f14c93bdc; no voters:
I20240501 14:10:27.008688 20374 raft_consensus.cc:2781] T dadab886b03540579d138cdb5b25e286 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:27.008747 20374 raft_consensus.cc:482] T dadab886b03540579d138cdb5b25e286 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:27.008773 20374 raft_consensus.cc:3037] T dadab886b03540579d138cdb5b25e286 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:27.009646 20374 raft_consensus.cc:504] T dadab886b03540579d138cdb5b25e286 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:27.009974 20374 leader_election.cc:288] T dadab886b03540579d138cdb5b25e286 P ec8cde17a00645e7b38d5d29f7425367 [CANDIDATE]: Term 1 election: Requested vote from peers 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829), fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847)
I20240501 14:10:27.010355 19375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dadab886b03540579d138cdb5b25e286" candidate_uuid: "ec8cde17a00645e7b38d5d29f7425367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc"
I20240501 14:10:27.010546 19375 raft_consensus.cc:3037] T dadab886b03540579d138cdb5b25e286 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:27.011466 19375 raft_consensus.cc:2454] T dadab886b03540579d138cdb5b25e286 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec8cde17a00645e7b38d5d29f7425367 in term 1.
I20240501 14:10:27.011684 19506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dadab886b03540579d138cdb5b25e286" candidate_uuid: "ec8cde17a00645e7b38d5d29f7425367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a0be6ef2144b7fb247ba9d3f051d66"
I20240501 14:10:27.011739 19571 leader_election.cc:302] T dadab886b03540579d138cdb5b25e286 P ec8cde17a00645e7b38d5d29f7425367 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ec8cde17a00645e7b38d5d29f7425367, fb1e6bc2d07b4ce3b622a66f14c93bdc; no voters:
I20240501 14:10:27.011811 19506 raft_consensus.cc:3037] T dadab886b03540579d138cdb5b25e286 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:27.011893 20374 raft_consensus.cc:2781] T dadab886b03540579d138cdb5b25e286 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:27.011986 20374 raft_consensus.cc:686] T dadab886b03540579d138cdb5b25e286 P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Becoming Leader. State: Replica: ec8cde17a00645e7b38d5d29f7425367, State: Running, Role: LEADER
I20240501 14:10:27.012123 20374 consensus_queue.cc:237] T dadab886b03540579d138cdb5b25e286 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:27.012610 19506 raft_consensus.cc:2454] T dadab886b03540579d138cdb5b25e286 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec8cde17a00645e7b38d5d29f7425367 in term 1.
I20240501 14:10:27.013742 19234 catalog_manager.cc:5484] T dadab886b03540579d138cdb5b25e286 P ec8cde17a00645e7b38d5d29f7425367 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131). New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } }
I20240501 14:10:27.126121 20378 consensus_queue.cc:1035] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:27.129243 19506 raft_consensus.cc:3037] T 661aaf83e3474966a257fcc039ef4784 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:27.130609 20378 consensus_queue.cc:1035] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:10:27.169708 20463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:10:27.169989 20463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:10:27.173103 20463 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 14:10:27.253019 19637 raft_consensus.cc:3037] T dab5e5722e2e4b0eaea73288d8dce265 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:27.253204 19506 raft_consensus.cc:1261] T dab5e5722e2e4b0eaea73288d8dce265 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:27.253899 20386 consensus_queue.cc:1035] T dab5e5722e2e4b0eaea73288d8dce265 P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:27.254181 19375 raft_consensus.cc:1261] T dadab886b03540579d138cdb5b25e286 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:27.254645 19637 raft_consensus.cc:1261] T dab5e5722e2e4b0eaea73288d8dce265 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:27.254730 20374 consensus_queue.cc:1035] T dadab886b03540579d138cdb5b25e286 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:27.259557 19507 raft_consensus.cc:1261] T dadab886b03540579d138cdb5b25e286 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:27.260586 20378 consensus_queue.cc:1035] T dadab886b03540579d138cdb5b25e286 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:27.260659 20386 consensus_queue.cc:1035] T dab5e5722e2e4b0eaea73288d8dce265 P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 94.8858 ms
time per row: 0.0474429 ms
I20240501 14:10:27.315443 19229 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45968:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:10:27.320904 19355 tablet_service.cc:1467] Processing CreateTablet for tablet 78e14dea8b6e4c0abd1c88dcd5b4e608 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6f6b0a378a484e5889d733ff982d9eb7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:27.320937 19485 tablet_service.cc:1467] Processing CreateTablet for tablet 78e14dea8b6e4c0abd1c88dcd5b4e608 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6f6b0a378a484e5889d733ff982d9eb7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:27.321014 19749 tablet_service.cc:1467] Processing CreateTablet for tablet 78e14dea8b6e4c0abd1c88dcd5b4e608 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6f6b0a378a484e5889d733ff982d9eb7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:27.321246 19355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78e14dea8b6e4c0abd1c88dcd5b4e608. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:27.321247 19485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78e14dea8b6e4c0abd1c88dcd5b4e608. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:27.321259 19749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78e14dea8b6e4c0abd1c88dcd5b4e608. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:27.323000 20518 tablet_bootstrap.cc:492] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Bootstrap starting.
I20240501 14:10:27.323824 19356 tablet_service.cc:1467] Processing CreateTablet for tablet 410a7dd0dd734684a82875912966657d (DEFAULT_TABLE table=rebalance_test_table_2 [id=6f6b0a378a484e5889d733ff982d9eb7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:27.323931 20518 tablet_bootstrap.cc:654] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:27.323885 19487 tablet_service.cc:1467] Processing CreateTablet for tablet 410a7dd0dd734684a82875912966657d (DEFAULT_TABLE table=rebalance_test_table_2 [id=6f6b0a378a484e5889d733ff982d9eb7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:27.323971 19618 tablet_service.cc:1467] Processing CreateTablet for tablet db0f01ee72b24ec9b84410cf94ef9759 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6f6b0a378a484e5889d733ff982d9eb7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:27.324077 19356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 410a7dd0dd734684a82875912966657d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:27.324121 19487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 410a7dd0dd734684a82875912966657d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:27.324223 19618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db0f01ee72b24ec9b84410cf94ef9759. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:27.325959 19355 tablet_service.cc:1467] Processing CreateTablet for tablet db0f01ee72b24ec9b84410cf94ef9759 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6f6b0a378a484e5889d733ff982d9eb7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:27.326157 20521 tablet_bootstrap.cc:492] T 410a7dd0dd734684a82875912966657d P 99a0be6ef2144b7fb247ba9d3f051d66: Bootstrap starting.
I20240501 14:10:27.326197 19355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db0f01ee72b24ec9b84410cf94ef9759. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:27.327227 20518 tablet_bootstrap.cc:492] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P fb1e6bc2d07b4ce3b622a66f14c93bdc: No bootstrap required, opened a new log
I20240501 14:10:27.327329 20518 ts_tablet_manager.cc:1367] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240501 14:10:27.327514 19748 tablet_service.cc:1467] Processing CreateTablet for tablet 410a7dd0dd734684a82875912966657d (DEFAULT_TABLE table=rebalance_test_table_2 [id=6f6b0a378a484e5889d733ff982d9eb7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:27.327754 19748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 410a7dd0dd734684a82875912966657d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:27.327798 20518 raft_consensus.cc:348] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:27.327919 20518 raft_consensus.cc:374] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:27.327992 20518 raft_consensus.cc:729] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1e6bc2d07b4ce3b622a66f14c93bdc, State: Initialized, Role: FOLLOWER
I20240501 14:10:27.328141 20518 consensus_queue.cc:260] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P fb1e6bc2d07b4ce3b622a66f14c93bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:27.328323 20522 tablet_bootstrap.cc:492] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367: Bootstrap starting.
I20240501 14:10:27.329046 20518 ts_tablet_manager.cc:1398] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:10:27.329208 20518 tablet_bootstrap.cc:492] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc: Bootstrap starting.
I20240501 14:10:27.329295 20522 tablet_bootstrap.cc:654] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:27.330072 20518 tablet_bootstrap.cc:654] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:27.330277 19749 tablet_service.cc:1467] Processing CreateTablet for tablet db0f01ee72b24ec9b84410cf94ef9759 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6f6b0a378a484e5889d733ff982d9eb7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:27.330497 19749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db0f01ee72b24ec9b84410cf94ef9759. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:27.330936 20521 tablet_bootstrap.cc:654] T 410a7dd0dd734684a82875912966657d P 99a0be6ef2144b7fb247ba9d3f051d66: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:27.331254 20519 tablet_bootstrap.cc:492] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P da3c4e948ff747678f18456c134d7ed4: Bootstrap starting.
I20240501 14:10:27.332152 20518 tablet_bootstrap.cc:492] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc: No bootstrap required, opened a new log
I20240501 14:10:27.332291 20518 ts_tablet_manager.cc:1367] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:10:27.332434 20519 tablet_bootstrap.cc:654] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P da3c4e948ff747678f18456c134d7ed4: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:27.333074 20518 raft_consensus.cc:348] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:27.333163 20518 raft_consensus.cc:374] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:27.333211 20518 raft_consensus.cc:729] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1e6bc2d07b4ce3b622a66f14c93bdc, State: Initialized, Role: FOLLOWER
I20240501 14:10:27.333235 20522 tablet_bootstrap.cc:492] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367: No bootstrap required, opened a new log
I20240501 14:10:27.333381 20522 ts_tablet_manager.cc:1367] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:10:27.333402 20518 consensus_queue.cc:260] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:27.333710 20518 ts_tablet_manager.cc:1398] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:10:27.333904 20518 tablet_bootstrap.cc:492] T db0f01ee72b24ec9b84410cf94ef9759 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Bootstrap starting.
I20240501 14:10:27.333946 20519 tablet_bootstrap.cc:492] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P da3c4e948ff747678f18456c134d7ed4: No bootstrap required, opened a new log
I20240501 14:10:27.333900 20522 raft_consensus.cc:348] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:27.334017 20522 raft_consensus.cc:374] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:27.334118 20519 ts_tablet_manager.cc:1367] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P da3c4e948ff747678f18456c134d7ed4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:10:27.334097 20522 raft_consensus.cc:729] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec8cde17a00645e7b38d5d29f7425367, State: Initialized, Role: FOLLOWER
I20240501 14:10:27.334297 20522 consensus_queue.cc:260] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:27.334659 20522 ts_tablet_manager.cc:1398] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:27.334831 20519 raft_consensus.cc:348] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:27.335002 20519 raft_consensus.cc:374] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:27.335130 20519 raft_consensus.cc:729] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3c4e948ff747678f18456c134d7ed4, State: Initialized, Role: FOLLOWER
I20240501 14:10:27.335350 20519 consensus_queue.cc:260] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P da3c4e948ff747678f18456c134d7ed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:27.336053 20518 tablet_bootstrap.cc:654] T db0f01ee72b24ec9b84410cf94ef9759 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:27.336407 20519 ts_tablet_manager.cc:1398] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P da3c4e948ff747678f18456c134d7ed4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:10:27.336642 20519 tablet_bootstrap.cc:492] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4: Bootstrap starting.
I20240501 14:10:27.337149 20521 tablet_bootstrap.cc:492] T 410a7dd0dd734684a82875912966657d P 99a0be6ef2144b7fb247ba9d3f051d66: No bootstrap required, opened a new log
I20240501 14:10:27.337273 20521 ts_tablet_manager.cc:1367] T 410a7dd0dd734684a82875912966657d P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 14:10:27.337323 20518 tablet_bootstrap.cc:492] T db0f01ee72b24ec9b84410cf94ef9759 P fb1e6bc2d07b4ce3b622a66f14c93bdc: No bootstrap required, opened a new log
I20240501 14:10:27.337433 20518 ts_tablet_manager.cc:1367] T db0f01ee72b24ec9b84410cf94ef9759 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240501 14:10:27.337740 20521 raft_consensus.cc:348] T 410a7dd0dd734684a82875912966657d P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:27.337872 20521 raft_consensus.cc:374] T 410a7dd0dd734684a82875912966657d P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:27.337921 20521 raft_consensus.cc:729] T 410a7dd0dd734684a82875912966657d P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a0be6ef2144b7fb247ba9d3f051d66, State: Initialized, Role: FOLLOWER
I20240501 14:10:27.337903 20518 raft_consensus.cc:348] T db0f01ee72b24ec9b84410cf94ef9759 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:27.338057 20521 consensus_queue.cc:260] T 410a7dd0dd734684a82875912966657d P 99a0be6ef2144b7fb247ba9d3f051d66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:27.338037 20518 raft_consensus.cc:374] T db0f01ee72b24ec9b84410cf94ef9759 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:27.338429 20521 ts_tablet_manager.cc:1398] T 410a7dd0dd734684a82875912966657d P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:27.338441 20518 raft_consensus.cc:729] T db0f01ee72b24ec9b84410cf94ef9759 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1e6bc2d07b4ce3b622a66f14c93bdc, State: Initialized, Role: FOLLOWER
I20240501 14:10:27.338593 20518 consensus_queue.cc:260] T db0f01ee72b24ec9b84410cf94ef9759 P fb1e6bc2d07b4ce3b622a66f14c93bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:27.338871 20521 tablet_bootstrap.cc:492] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P 99a0be6ef2144b7fb247ba9d3f051d66: Bootstrap starting.
I20240501 14:10:27.339499 20518 ts_tablet_manager.cc:1398] T db0f01ee72b24ec9b84410cf94ef9759 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:10:27.339893 20521 tablet_bootstrap.cc:654] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P 99a0be6ef2144b7fb247ba9d3f051d66: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:27.340994 20521 tablet_bootstrap.cc:492] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P 99a0be6ef2144b7fb247ba9d3f051d66: No bootstrap required, opened a new log
I20240501 14:10:27.341849 20521 ts_tablet_manager.cc:1367] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:10:27.341974 20519 tablet_bootstrap.cc:654] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:27.342350 20521 raft_consensus.cc:348] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:27.342741 20521 raft_consensus.cc:374] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:27.342810 20521 raft_consensus.cc:729] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a0be6ef2144b7fb247ba9d3f051d66, State: Initialized, Role: FOLLOWER
I20240501 14:10:27.342945 20521 consensus_queue.cc:260] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P 99a0be6ef2144b7fb247ba9d3f051d66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:27.343103 20519 tablet_bootstrap.cc:492] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4: No bootstrap required, opened a new log
I20240501 14:10:27.343194 20519 ts_tablet_manager.cc:1367] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:10:27.343379 20521 ts_tablet_manager.cc:1398] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:27.343600 20519 raft_consensus.cc:348] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:27.343693 20519 raft_consensus.cc:374] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:27.343722 20519 raft_consensus.cc:729] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3c4e948ff747678f18456c134d7ed4, State: Initialized, Role: FOLLOWER
I20240501 14:10:27.343839 20519 consensus_queue.cc:260] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:27.344072 20519 ts_tablet_manager.cc:1398] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:27.344242 20519 tablet_bootstrap.cc:492] T db0f01ee72b24ec9b84410cf94ef9759 P da3c4e948ff747678f18456c134d7ed4: Bootstrap starting.
I20240501 14:10:27.345101 20519 tablet_bootstrap.cc:654] T db0f01ee72b24ec9b84410cf94ef9759 P da3c4e948ff747678f18456c134d7ed4: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:27.346364 20519 tablet_bootstrap.cc:492] T db0f01ee72b24ec9b84410cf94ef9759 P da3c4e948ff747678f18456c134d7ed4: No bootstrap required, opened a new log
I20240501 14:10:27.346468 20519 ts_tablet_manager.cc:1367] T db0f01ee72b24ec9b84410cf94ef9759 P da3c4e948ff747678f18456c134d7ed4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:10:27.346905 20519 raft_consensus.cc:348] T db0f01ee72b24ec9b84410cf94ef9759 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:27.347016 20519 raft_consensus.cc:374] T db0f01ee72b24ec9b84410cf94ef9759 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:27.347090 20519 raft_consensus.cc:729] T db0f01ee72b24ec9b84410cf94ef9759 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3c4e948ff747678f18456c134d7ed4, State: Initialized, Role: FOLLOWER
I20240501 14:10:27.347213 20519 consensus_queue.cc:260] T db0f01ee72b24ec9b84410cf94ef9759 P da3c4e948ff747678f18456c134d7ed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:27.347504 20519 ts_tablet_manager.cc:1398] T db0f01ee72b24ec9b84410cf94ef9759 P da3c4e948ff747678f18456c134d7ed4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:27.418859 20511 raft_consensus.cc:482] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:27.419013 20511 raft_consensus.cc:504] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:27.420110 20511 leader_election.cc:288] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P fb1e6bc2d07b4ce3b622a66f14c93bdc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829), da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357)
I20240501 14:10:27.424693 19506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78e14dea8b6e4c0abd1c88dcd5b4e608" candidate_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" is_pre_election: true
I20240501 14:10:27.424692 19769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78e14dea8b6e4c0abd1c88dcd5b4e608" candidate_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3c4e948ff747678f18456c134d7ed4" is_pre_election: true
I20240501 14:10:27.424835 19506 raft_consensus.cc:2454] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb1e6bc2d07b4ce3b622a66f14c93bdc in term 0.
I20240501 14:10:27.424844 19769 raft_consensus.cc:2454] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb1e6bc2d07b4ce3b622a66f14c93bdc in term 0.
I20240501 14:10:27.425202 19310 leader_election.cc:302] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P fb1e6bc2d07b4ce3b622a66f14c93bdc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99a0be6ef2144b7fb247ba9d3f051d66, fb1e6bc2d07b4ce3b622a66f14c93bdc; no voters:
I20240501 14:10:27.425442 20511 raft_consensus.cc:2781] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:27.425501 20511 raft_consensus.cc:482] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:27.425527 20511 raft_consensus.cc:3037] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:27.426677 20511 raft_consensus.cc:504] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:27.427016 20511 leader_election.cc:288] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P fb1e6bc2d07b4ce3b622a66f14c93bdc [CANDIDATE]: Term 1 election: Requested vote from peers 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829), da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357)
I20240501 14:10:27.427347 19506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78e14dea8b6e4c0abd1c88dcd5b4e608" candidate_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a0be6ef2144b7fb247ba9d3f051d66"
I20240501 14:10:27.427379 19769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78e14dea8b6e4c0abd1c88dcd5b4e608" candidate_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3c4e948ff747678f18456c134d7ed4"
I20240501 14:10:27.427471 19506 raft_consensus.cc:3037] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:27.427486 19769 raft_consensus.cc:3037] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:27.428213 19506 raft_consensus.cc:2454] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb1e6bc2d07b4ce3b622a66f14c93bdc in term 1.
I20240501 14:10:27.428248 19769 raft_consensus.cc:2454] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb1e6bc2d07b4ce3b622a66f14c93bdc in term 1.
I20240501 14:10:27.428505 19310 leader_election.cc:302] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P fb1e6bc2d07b4ce3b622a66f14c93bdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99a0be6ef2144b7fb247ba9d3f051d66, fb1e6bc2d07b4ce3b622a66f14c93bdc; no voters:
I20240501 14:10:27.428673 20511 raft_consensus.cc:2781] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:27.428778 20511 raft_consensus.cc:686] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Becoming Leader. State: Replica: fb1e6bc2d07b4ce3b622a66f14c93bdc, State: Running, Role: LEADER
I20240501 14:10:27.429080 20511 consensus_queue.cc:237] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:27.431696 19229 catalog_manager.cc:5484] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P fb1e6bc2d07b4ce3b622a66f14c93bdc reported cstate change: term changed from 0 to 1, leader changed from <none> to fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129). New cstate: current_term: 1 leader_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } }
I20240501 14:10:27.480254 20511 raft_consensus.cc:482] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:27.480387 20511 raft_consensus.cc:504] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:27.480825 20511 leader_election.cc:288] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357), 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829)
I20240501 14:10:27.481274 19506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "410a7dd0dd734684a82875912966657d" candidate_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" is_pre_election: true
I20240501 14:10:27.481280 19769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "410a7dd0dd734684a82875912966657d" candidate_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3c4e948ff747678f18456c134d7ed4" is_pre_election: true
I20240501 14:10:27.481413 19506 raft_consensus.cc:2454] T 410a7dd0dd734684a82875912966657d P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb1e6bc2d07b4ce3b622a66f14c93bdc in term 0.
I20240501 14:10:27.481417 19769 raft_consensus.cc:2454] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb1e6bc2d07b4ce3b622a66f14c93bdc in term 0.
I20240501 14:10:27.481710 19311 leader_election.cc:302] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da3c4e948ff747678f18456c134d7ed4, fb1e6bc2d07b4ce3b622a66f14c93bdc; no voters:
I20240501 14:10:27.481870 20511 raft_consensus.cc:2781] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:27.481930 20511 raft_consensus.cc:482] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:27.481964 20511 raft_consensus.cc:3037] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:27.482738 20511 raft_consensus.cc:504] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:27.483029 20511 leader_election.cc:288] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [CANDIDATE]: Term 1 election: Requested vote from peers da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357), 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829)
I20240501 14:10:27.483357 19769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "410a7dd0dd734684a82875912966657d" candidate_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3c4e948ff747678f18456c134d7ed4"
I20240501 14:10:27.483361 19506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "410a7dd0dd734684a82875912966657d" candidate_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a0be6ef2144b7fb247ba9d3f051d66"
I20240501 14:10:27.483467 19769 raft_consensus.cc:3037] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:27.483482 19506 raft_consensus.cc:3037] T 410a7dd0dd734684a82875912966657d P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:27.484192 19506 raft_consensus.cc:2454] T 410a7dd0dd734684a82875912966657d P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb1e6bc2d07b4ce3b622a66f14c93bdc in term 1.
I20240501 14:10:27.484205 19769 raft_consensus.cc:2454] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb1e6bc2d07b4ce3b622a66f14c93bdc in term 1.
I20240501 14:10:27.484437 19310 leader_election.cc:302] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99a0be6ef2144b7fb247ba9d3f051d66, fb1e6bc2d07b4ce3b622a66f14c93bdc; no voters:
I20240501 14:10:27.484586 20511 raft_consensus.cc:2781] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:27.484702 20511 raft_consensus.cc:686] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Becoming Leader. State: Replica: fb1e6bc2d07b4ce3b622a66f14c93bdc, State: Running, Role: LEADER
I20240501 14:10:27.484843 20511 consensus_queue.cc:237] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:27.486459 19229 catalog_manager.cc:5484] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc reported cstate change: term changed from 0 to 1, leader changed from <none> to fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129). New cstate: current_term: 1 leader_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } }
I20240501 14:10:27.565646 20496 raft_consensus.cc:482] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:27.565801 20496 raft_consensus.cc:504] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:27.566249 20496 leader_election.cc:288] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847), da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357)
I20240501 14:10:27.566725 19769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db0f01ee72b24ec9b84410cf94ef9759" candidate_uuid: "ec8cde17a00645e7b38d5d29f7425367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3c4e948ff747678f18456c134d7ed4" is_pre_election: true
I20240501 14:10:27.566756 19375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db0f01ee72b24ec9b84410cf94ef9759" candidate_uuid: "ec8cde17a00645e7b38d5d29f7425367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" is_pre_election: true
I20240501 14:10:27.566875 19769 raft_consensus.cc:2454] T db0f01ee72b24ec9b84410cf94ef9759 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec8cde17a00645e7b38d5d29f7425367 in term 0.
I20240501 14:10:27.566901 19375 raft_consensus.cc:2454] T db0f01ee72b24ec9b84410cf94ef9759 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec8cde17a00645e7b38d5d29f7425367 in term 0.
I20240501 14:10:27.567185 19573 leader_election.cc:302] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da3c4e948ff747678f18456c134d7ed4, ec8cde17a00645e7b38d5d29f7425367; no voters:
I20240501 14:10:27.567348 20496 raft_consensus.cc:2781] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:27.567409 20496 raft_consensus.cc:482] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:27.567452 20496 raft_consensus.cc:3037] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:27.568272 20496 raft_consensus.cc:504] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:27.568579 20496 leader_election.cc:288] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 [CANDIDATE]: Term 1 election: Requested vote from peers fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847), da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357)
I20240501 14:10:27.568890 19375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db0f01ee72b24ec9b84410cf94ef9759" candidate_uuid: "ec8cde17a00645e7b38d5d29f7425367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc"
I20240501 14:10:27.568898 19769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db0f01ee72b24ec9b84410cf94ef9759" candidate_uuid: "ec8cde17a00645e7b38d5d29f7425367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3c4e948ff747678f18456c134d7ed4"
I20240501 14:10:27.569001 19769 raft_consensus.cc:3037] T db0f01ee72b24ec9b84410cf94ef9759 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:27.569005 19375 raft_consensus.cc:3037] T db0f01ee72b24ec9b84410cf94ef9759 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:27.569721 19769 raft_consensus.cc:2454] T db0f01ee72b24ec9b84410cf94ef9759 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec8cde17a00645e7b38d5d29f7425367 in term 1.
I20240501 14:10:27.569814 19375 raft_consensus.cc:2454] T db0f01ee72b24ec9b84410cf94ef9759 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec8cde17a00645e7b38d5d29f7425367 in term 1.
I20240501 14:10:27.569981 19573 leader_election.cc:302] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da3c4e948ff747678f18456c134d7ed4, ec8cde17a00645e7b38d5d29f7425367; no voters:
I20240501 14:10:27.570135 20496 raft_consensus.cc:2781] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:27.570232 20496 raft_consensus.cc:686] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Becoming Leader. State: Replica: ec8cde17a00645e7b38d5d29f7425367, State: Running, Role: LEADER
I20240501 14:10:27.570416 20496 consensus_queue.cc:237] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:27.572010 19229 catalog_manager.cc:5484] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131). New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } }
W20240501 14:10:27.727618 20537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:10:27.727861 20537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:10:27.731073 20537 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240501 14:10:27.820220 19769 raft_consensus.cc:1261] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:27.820470 19506 raft_consensus.cc:1261] T 410a7dd0dd734684a82875912966657d P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:27.821159 20511 consensus_queue.cc:1035] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:27.821255 19506 raft_consensus.cc:1261] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:27.821470 20499 consensus_queue.cc:1035] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:27.821990 19768 raft_consensus.cc:1261] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:27.823776 19375 raft_consensus.cc:1261] T db0f01ee72b24ec9b84410cf94ef9759 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:27.824137 20496 consensus_queue.cc:1035] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:27.825160 19769 raft_consensus.cc:1261] T db0f01ee72b24ec9b84410cf94ef9759 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:27.825361 20499 consensus_queue.cc:1035] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:27.825695 20499 consensus_queue.cc:1035] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:27.826745 20496 consensus_queue.cc:1035] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 112.787 ms
time per row: 0.0563933 ms
I20240501 14:10:27.884965 19232 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45968:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:10:27.890543 19355 tablet_service.cc:1467] Processing CreateTablet for tablet ffb4b6a2c19b497f8117fbbc9ea50082 (DEFAULT_TABLE table=rebalance_test_table_3 [id=35d6f8fac5394197b987aa44dab206c1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:27.890707 19485 tablet_service.cc:1467] Processing CreateTablet for tablet ffb4b6a2c19b497f8117fbbc9ea50082 (DEFAULT_TABLE table=rebalance_test_table_3 [id=35d6f8fac5394197b987aa44dab206c1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:27.890868 19355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffb4b6a2c19b497f8117fbbc9ea50082. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:27.891515 19618 tablet_service.cc:1467] Processing CreateTablet for tablet ffb4b6a2c19b497f8117fbbc9ea50082 (DEFAULT_TABLE table=rebalance_test_table_3 [id=35d6f8fac5394197b987aa44dab206c1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:27.891748 19618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffb4b6a2c19b497f8117fbbc9ea50082. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:27.892783 20596 tablet_bootstrap.cc:492] T ffb4b6a2c19b497f8117fbbc9ea50082 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Bootstrap starting.
I20240501 14:10:27.893646 19355 tablet_service.cc:1467] Processing CreateTablet for tablet ec2a46f8cfb3406082970e5fdb59b101 (DEFAULT_TABLE table=rebalance_test_table_3 [id=35d6f8fac5394197b987aa44dab206c1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:27.893690 20597 tablet_bootstrap.cc:492] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367: Bootstrap starting.
I20240501 14:10:27.893745 20596 tablet_bootstrap.cc:654] T ffb4b6a2c19b497f8117fbbc9ea50082 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:27.893764 19749 tablet_service.cc:1467] Processing CreateTablet for tablet b6be9a115fe14c098c29cad819849f95 (DEFAULT_TABLE table=rebalance_test_table_3 [id=35d6f8fac5394197b987aa44dab206c1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:27.893875 19355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec2a46f8cfb3406082970e5fdb59b101. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:27.894043 19749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6be9a115fe14c098c29cad819849f95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:27.894610 20597 tablet_bootstrap.cc:654] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:27.895722 20597 tablet_bootstrap.cc:492] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367: No bootstrap required, opened a new log
I20240501 14:10:27.895879 20597 ts_tablet_manager.cc:1367] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:10:27.896399 20600 tablet_bootstrap.cc:492] T b6be9a115fe14c098c29cad819849f95 P da3c4e948ff747678f18456c134d7ed4: Bootstrap starting.
I20240501 14:10:27.896345 20597 raft_consensus.cc:348] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:27.896457 20597 raft_consensus.cc:374] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:27.896525 20597 raft_consensus.cc:729] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec8cde17a00645e7b38d5d29f7425367, State: Initialized, Role: FOLLOWER
I20240501 14:10:27.896628 19748 tablet_service.cc:1467] Processing CreateTablet for tablet ec2a46f8cfb3406082970e5fdb59b101 (DEFAULT_TABLE table=rebalance_test_table_3 [id=35d6f8fac5394197b987aa44dab206c1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:27.896675 20597 consensus_queue.cc:260] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:27.896862 19748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec2a46f8cfb3406082970e5fdb59b101. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:27.896988 20597 ts_tablet_manager.cc:1398] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:27.897213 20600 tablet_bootstrap.cc:654] T b6be9a115fe14c098c29cad819849f95 P da3c4e948ff747678f18456c134d7ed4: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:27.898049 20600 tablet_bootstrap.cc:492] T b6be9a115fe14c098c29cad819849f95 P da3c4e948ff747678f18456c134d7ed4: No bootstrap required, opened a new log
I20240501 14:10:27.898133 20600 ts_tablet_manager.cc:1367] T b6be9a115fe14c098c29cad819849f95 P da3c4e948ff747678f18456c134d7ed4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:10:27.898548 20600 raft_consensus.cc:348] T b6be9a115fe14c098c29cad819849f95 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:27.898702 20600 raft_consensus.cc:374] T b6be9a115fe14c098c29cad819849f95 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:27.898757 20600 raft_consensus.cc:729] T b6be9a115fe14c098c29cad819849f95 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3c4e948ff747678f18456c134d7ed4, State: Initialized, Role: FOLLOWER
I20240501 14:10:27.898887 20600 consensus_queue.cc:260] T b6be9a115fe14c098c29cad819849f95 P da3c4e948ff747678f18456c134d7ed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:27.899346 19485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffb4b6a2c19b497f8117fbbc9ea50082. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:27.899492 19617 tablet_service.cc:1467] Processing CreateTablet for tablet ec2a46f8cfb3406082970e5fdb59b101 (DEFAULT_TABLE table=rebalance_test_table_3 [id=35d6f8fac5394197b987aa44dab206c1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:27.899756 19617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec2a46f8cfb3406082970e5fdb59b101. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:27.899334 19487 tablet_service.cc:1467] Processing CreateTablet for tablet b6be9a115fe14c098c29cad819849f95 (DEFAULT_TABLE table=rebalance_test_table_3 [id=35d6f8fac5394197b987aa44dab206c1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:27.900060 19487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6be9a115fe14c098c29cad819849f95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:27.899221 20600 ts_tablet_manager.cc:1398] T b6be9a115fe14c098c29cad819849f95 P da3c4e948ff747678f18456c134d7ed4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:27.901742 20597 tablet_bootstrap.cc:492] T ec2a46f8cfb3406082970e5fdb59b101 P ec8cde17a00645e7b38d5d29f7425367: Bootstrap starting.
I20240501 14:10:27.901856 20600 tablet_bootstrap.cc:492] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4: Bootstrap starting.
I20240501 14:10:27.902323 19618 tablet_service.cc:1467] Processing CreateTablet for tablet b6be9a115fe14c098c29cad819849f95 (DEFAULT_TABLE table=rebalance_test_table_3 [id=35d6f8fac5394197b987aa44dab206c1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:27.902515 20597 tablet_bootstrap.cc:654] T ec2a46f8cfb3406082970e5fdb59b101 P ec8cde17a00645e7b38d5d29f7425367: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:27.902590 19618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6be9a115fe14c098c29cad819849f95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:27.902630 20600 tablet_bootstrap.cc:654] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:27.903688 20597 tablet_bootstrap.cc:492] T ec2a46f8cfb3406082970e5fdb59b101 P ec8cde17a00645e7b38d5d29f7425367: No bootstrap required, opened a new log
I20240501 14:10:27.903697 20596 tablet_bootstrap.cc:492] T ffb4b6a2c19b497f8117fbbc9ea50082 P fb1e6bc2d07b4ce3b622a66f14c93bdc: No bootstrap required, opened a new log
I20240501 14:10:27.903775 20596 ts_tablet_manager.cc:1367] T ffb4b6a2c19b497f8117fbbc9ea50082 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20240501 14:10:27.903774 20597 ts_tablet_manager.cc:1367] T ec2a46f8cfb3406082970e5fdb59b101 P ec8cde17a00645e7b38d5d29f7425367: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:10:27.904225 20597 raft_consensus.cc:348] T ec2a46f8cfb3406082970e5fdb59b101 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:27.904203 20596 raft_consensus.cc:348] T ffb4b6a2c19b497f8117fbbc9ea50082 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:27.904315 20596 raft_consensus.cc:374] T ffb4b6a2c19b497f8117fbbc9ea50082 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:27.904315 20597 raft_consensus.cc:374] T ec2a46f8cfb3406082970e5fdb59b101 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:27.904348 20596 raft_consensus.cc:729] T ffb4b6a2c19b497f8117fbbc9ea50082 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1e6bc2d07b4ce3b622a66f14c93bdc, State: Initialized, Role: FOLLOWER
I20240501 14:10:27.904357 20597 raft_consensus.cc:729] T ec2a46f8cfb3406082970e5fdb59b101 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec8cde17a00645e7b38d5d29f7425367, State: Initialized, Role: FOLLOWER
I20240501 14:10:27.904476 20596 consensus_queue.cc:260] T ffb4b6a2c19b497f8117fbbc9ea50082 P fb1e6bc2d07b4ce3b622a66f14c93bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:27.904495 20597 consensus_queue.cc:260] T ec2a46f8cfb3406082970e5fdb59b101 P ec8cde17a00645e7b38d5d29f7425367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:27.904752 20596 ts_tablet_manager.cc:1398] T ffb4b6a2c19b497f8117fbbc9ea50082 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:10:27.904817 20597 ts_tablet_manager.cc:1398] T ec2a46f8cfb3406082970e5fdb59b101 P ec8cde17a00645e7b38d5d29f7425367: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:27.904917 20596 tablet_bootstrap.cc:492] T ec2a46f8cfb3406082970e5fdb59b101 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Bootstrap starting.
I20240501 14:10:27.904951 20597 tablet_bootstrap.cc:492] T b6be9a115fe14c098c29cad819849f95 P ec8cde17a00645e7b38d5d29f7425367: Bootstrap starting.
I20240501 14:10:27.906102 20597 tablet_bootstrap.cc:654] T b6be9a115fe14c098c29cad819849f95 P ec8cde17a00645e7b38d5d29f7425367: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:27.906102 20596 tablet_bootstrap.cc:654] T ec2a46f8cfb3406082970e5fdb59b101 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:27.907263 20602 tablet_bootstrap.cc:492] T ffb4b6a2c19b497f8117fbbc9ea50082 P 99a0be6ef2144b7fb247ba9d3f051d66: Bootstrap starting.
I20240501 14:10:27.908133 20602 tablet_bootstrap.cc:654] T ffb4b6a2c19b497f8117fbbc9ea50082 P 99a0be6ef2144b7fb247ba9d3f051d66: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:27.909181 20596 tablet_bootstrap.cc:492] T ec2a46f8cfb3406082970e5fdb59b101 P fb1e6bc2d07b4ce3b622a66f14c93bdc: No bootstrap required, opened a new log
I20240501 14:10:27.909263 20596 ts_tablet_manager.cc:1367] T ec2a46f8cfb3406082970e5fdb59b101 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:10:27.909420 20600 tablet_bootstrap.cc:492] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4: No bootstrap required, opened a new log
I20240501 14:10:27.909518 20600 ts_tablet_manager.cc:1367] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240501 14:10:27.909644 20597 tablet_bootstrap.cc:492] T b6be9a115fe14c098c29cad819849f95 P ec8cde17a00645e7b38d5d29f7425367: No bootstrap required, opened a new log
I20240501 14:10:27.909729 20597 ts_tablet_manager.cc:1367] T b6be9a115fe14c098c29cad819849f95 P ec8cde17a00645e7b38d5d29f7425367: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:10:27.909673 20596 raft_consensus.cc:348] T ec2a46f8cfb3406082970e5fdb59b101 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:27.909775 20596 raft_consensus.cc:374] T ec2a46f8cfb3406082970e5fdb59b101 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:27.909813 20596 raft_consensus.cc:729] T ec2a46f8cfb3406082970e5fdb59b101 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1e6bc2d07b4ce3b622a66f14c93bdc, State: Initialized, Role: FOLLOWER
I20240501 14:10:27.909946 20596 consensus_queue.cc:260] T ec2a46f8cfb3406082970e5fdb59b101 P fb1e6bc2d07b4ce3b622a66f14c93bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:27.910029 20600 raft_consensus.cc:348] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:27.910153 20600 raft_consensus.cc:374] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:27.910192 20600 raft_consensus.cc:729] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3c4e948ff747678f18456c134d7ed4, State: Initialized, Role: FOLLOWER
I20240501 14:10:27.910219 20596 ts_tablet_manager.cc:1398] T ec2a46f8cfb3406082970e5fdb59b101 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:10:27.910312 20600 consensus_queue.cc:260] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:27.910295 20597 raft_consensus.cc:348] T b6be9a115fe14c098c29cad819849f95 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:27.910450 20597 raft_consensus.cc:374] T b6be9a115fe14c098c29cad819849f95 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:27.910485 20597 raft_consensus.cc:729] T b6be9a115fe14c098c29cad819849f95 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec8cde17a00645e7b38d5d29f7425367, State: Initialized, Role: FOLLOWER
I20240501 14:10:27.910604 20600 ts_tablet_manager.cc:1398] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240501 14:10:27.910621 20597 consensus_queue.cc:260] T b6be9a115fe14c098c29cad819849f95 P ec8cde17a00645e7b38d5d29f7425367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:27.910725 20602 tablet_bootstrap.cc:492] T ffb4b6a2c19b497f8117fbbc9ea50082 P 99a0be6ef2144b7fb247ba9d3f051d66: No bootstrap required, opened a new log
I20240501 14:10:27.910816 20602 ts_tablet_manager.cc:1367] T ffb4b6a2c19b497f8117fbbc9ea50082 P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:10:27.910974 20597 ts_tablet_manager.cc:1398] T b6be9a115fe14c098c29cad819849f95 P ec8cde17a00645e7b38d5d29f7425367: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:27.911448 20602 raft_consensus.cc:348] T ffb4b6a2c19b497f8117fbbc9ea50082 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:27.911556 20602 raft_consensus.cc:374] T ffb4b6a2c19b497f8117fbbc9ea50082 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:27.911602 20602 raft_consensus.cc:729] T ffb4b6a2c19b497f8117fbbc9ea50082 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a0be6ef2144b7fb247ba9d3f051d66, State: Initialized, Role: FOLLOWER
I20240501 14:10:27.911729 20602 consensus_queue.cc:260] T ffb4b6a2c19b497f8117fbbc9ea50082 P 99a0be6ef2144b7fb247ba9d3f051d66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:27.912014 20602 ts_tablet_manager.cc:1398] T ffb4b6a2c19b497f8117fbbc9ea50082 P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:10:27.912164 20602 tablet_bootstrap.cc:492] T b6be9a115fe14c098c29cad819849f95 P 99a0be6ef2144b7fb247ba9d3f051d66: Bootstrap starting.
I20240501 14:10:27.913010 20602 tablet_bootstrap.cc:654] T b6be9a115fe14c098c29cad819849f95 P 99a0be6ef2144b7fb247ba9d3f051d66: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:27.914117 20602 tablet_bootstrap.cc:492] T b6be9a115fe14c098c29cad819849f95 P 99a0be6ef2144b7fb247ba9d3f051d66: No bootstrap required, opened a new log
I20240501 14:10:27.914206 20602 ts_tablet_manager.cc:1367] T b6be9a115fe14c098c29cad819849f95 P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:10:27.914659 20602 raft_consensus.cc:348] T b6be9a115fe14c098c29cad819849f95 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:27.914762 20602 raft_consensus.cc:374] T b6be9a115fe14c098c29cad819849f95 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:27.914804 20602 raft_consensus.cc:729] T b6be9a115fe14c098c29cad819849f95 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a0be6ef2144b7fb247ba9d3f051d66, State: Initialized, Role: FOLLOWER
I20240501 14:10:27.914920 20602 consensus_queue.cc:260] T b6be9a115fe14c098c29cad819849f95 P 99a0be6ef2144b7fb247ba9d3f051d66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:27.915185 20602 ts_tablet_manager.cc:1398] T b6be9a115fe14c098c29cad819849f95 P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:27.923786 20378 raft_consensus.cc:482] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:27.923882 20378 raft_consensus.cc:504] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:27.924252 20378 leader_election.cc:288] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847), 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829)
I20240501 14:10:27.924593 19375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffb4b6a2c19b497f8117fbbc9ea50082" candidate_uuid: "ec8cde17a00645e7b38d5d29f7425367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" is_pre_election: true
I20240501 14:10:27.924629 19506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffb4b6a2c19b497f8117fbbc9ea50082" candidate_uuid: "ec8cde17a00645e7b38d5d29f7425367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" is_pre_election: true
I20240501 14:10:27.924743 19506 raft_consensus.cc:2454] T ffb4b6a2c19b497f8117fbbc9ea50082 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec8cde17a00645e7b38d5d29f7425367 in term 0.
I20240501 14:10:27.924746 19375 raft_consensus.cc:2454] T ffb4b6a2c19b497f8117fbbc9ea50082 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec8cde17a00645e7b38d5d29f7425367 in term 0.
I20240501 14:10:27.924983 19572 leader_election.cc:302] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99a0be6ef2144b7fb247ba9d3f051d66, ec8cde17a00645e7b38d5d29f7425367; no voters:
I20240501 14:10:27.925220 20378 raft_consensus.cc:2781] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:27.925273 20378 raft_consensus.cc:482] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:27.925316 20378 raft_consensus.cc:3037] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:27.926515 20378 raft_consensus.cc:504] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:27.926810 20378 leader_election.cc:288] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 [CANDIDATE]: Term 1 election: Requested vote from peers fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847), 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829)
I20240501 14:10:27.927130 19375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffb4b6a2c19b497f8117fbbc9ea50082" candidate_uuid: "ec8cde17a00645e7b38d5d29f7425367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc"
I20240501 14:10:27.927242 19375 raft_consensus.cc:3037] T ffb4b6a2c19b497f8117fbbc9ea50082 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:27.927291 19506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffb4b6a2c19b497f8117fbbc9ea50082" candidate_uuid: "ec8cde17a00645e7b38d5d29f7425367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a0be6ef2144b7fb247ba9d3f051d66"
I20240501 14:10:27.927414 19506 raft_consensus.cc:3037] T ffb4b6a2c19b497f8117fbbc9ea50082 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:27.928277 19375 raft_consensus.cc:2454] T ffb4b6a2c19b497f8117fbbc9ea50082 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec8cde17a00645e7b38d5d29f7425367 in term 1.
I20240501 14:10:27.928445 19506 raft_consensus.cc:2454] T ffb4b6a2c19b497f8117fbbc9ea50082 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec8cde17a00645e7b38d5d29f7425367 in term 1.
I20240501 14:10:27.928601 19571 leader_election.cc:302] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ec8cde17a00645e7b38d5d29f7425367, fb1e6bc2d07b4ce3b622a66f14c93bdc; no voters:
I20240501 14:10:27.928781 20378 raft_consensus.cc:2781] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:27.928885 20378 raft_consensus.cc:686] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Becoming Leader. State: Replica: ec8cde17a00645e7b38d5d29f7425367, State: Running, Role: LEADER
I20240501 14:10:27.929023 20378 consensus_queue.cc:237] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:27.930733 19233 catalog_manager.cc:5484] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131). New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } }
I20240501 14:10:28.001282 20386 raft_consensus.cc:482] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:28.001420 20386 raft_consensus.cc:504] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:28.001830 20386 leader_election.cc:288] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847), ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201)
I20240501 14:10:28.002252 19637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec2a46f8cfb3406082970e5fdb59b101" candidate_uuid: "da3c4e948ff747678f18456c134d7ed4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec8cde17a00645e7b38d5d29f7425367" is_pre_election: true
I20240501 14:10:28.002308 19375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec2a46f8cfb3406082970e5fdb59b101" candidate_uuid: "da3c4e948ff747678f18456c134d7ed4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" is_pre_election: true
I20240501 14:10:28.002399 19637 raft_consensus.cc:2454] T ec2a46f8cfb3406082970e5fdb59b101 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da3c4e948ff747678f18456c134d7ed4 in term 0.
I20240501 14:10:28.002483 19375 raft_consensus.cc:2454] T ec2a46f8cfb3406082970e5fdb59b101 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da3c4e948ff747678f18456c134d7ed4 in term 0.
I20240501 14:10:28.002760 19702 leader_election.cc:302] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da3c4e948ff747678f18456c134d7ed4, fb1e6bc2d07b4ce3b622a66f14c93bdc; no voters:
I20240501 14:10:28.002952 20386 raft_consensus.cc:2781] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:28.003012 20386 raft_consensus.cc:482] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:28.003099 20386 raft_consensus.cc:3037] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:28.003978 20386 raft_consensus.cc:504] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:28.004287 20386 leader_election.cc:288] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4 [CANDIDATE]: Term 1 election: Requested vote from peers fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847), ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201)
I20240501 14:10:28.004624 19375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec2a46f8cfb3406082970e5fdb59b101" candidate_uuid: "da3c4e948ff747678f18456c134d7ed4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc"
I20240501 14:10:28.004632 19637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec2a46f8cfb3406082970e5fdb59b101" candidate_uuid: "da3c4e948ff747678f18456c134d7ed4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec8cde17a00645e7b38d5d29f7425367"
I20240501 14:10:28.004737 19375 raft_consensus.cc:3037] T ec2a46f8cfb3406082970e5fdb59b101 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:28.004782 19637 raft_consensus.cc:3037] T ec2a46f8cfb3406082970e5fdb59b101 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:28.005513 19637 raft_consensus.cc:2454] T ec2a46f8cfb3406082970e5fdb59b101 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da3c4e948ff747678f18456c134d7ed4 in term 1.
I20240501 14:10:28.005519 19375 raft_consensus.cc:2454] T ec2a46f8cfb3406082970e5fdb59b101 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da3c4e948ff747678f18456c134d7ed4 in term 1.
I20240501 14:10:28.005798 19703 leader_election.cc:302] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da3c4e948ff747678f18456c134d7ed4, ec8cde17a00645e7b38d5d29f7425367; no voters:
I20240501 14:10:28.005968 20386 raft_consensus.cc:2781] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:28.006163 20386 raft_consensus.cc:686] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4 [term 1 LEADER]: Becoming Leader. State: Replica: da3c4e948ff747678f18456c134d7ed4, State: Running, Role: LEADER
I20240501 14:10:28.006310 20386 consensus_queue.cc:237] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:28.007869 19233 catalog_manager.cc:5484] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4 reported cstate change: term changed from 0 to 1, leader changed from <none> to da3c4e948ff747678f18456c134d7ed4 (127.7.39.132). New cstate: current_term: 1 leader_uuid: "da3c4e948ff747678f18456c134d7ed4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } }
I20240501 14:10:28.031497 20372 raft_consensus.cc:482] T b6be9a115fe14c098c29cad819849f95 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:28.031649 20372 raft_consensus.cc:504] T b6be9a115fe14c098c29cad819849f95 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:28.032156 20372 leader_election.cc:288] T b6be9a115fe14c098c29cad819849f95 P 99a0be6ef2144b7fb247ba9d3f051d66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201), da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357)
I20240501 14:10:28.032692 19637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6be9a115fe14c098c29cad819849f95" candidate_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec8cde17a00645e7b38d5d29f7425367" is_pre_election: true
I20240501 14:10:28.032836 19637 raft_consensus.cc:2454] T b6be9a115fe14c098c29cad819849f95 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99a0be6ef2144b7fb247ba9d3f051d66 in term 0.
I20240501 14:10:28.033167 19441 leader_election.cc:302] T b6be9a115fe14c098c29cad819849f95 P 99a0be6ef2144b7fb247ba9d3f051d66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99a0be6ef2144b7fb247ba9d3f051d66, ec8cde17a00645e7b38d5d29f7425367; no voters:
I20240501 14:10:28.033350 20372 raft_consensus.cc:2781] T b6be9a115fe14c098c29cad819849f95 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:28.033403 20372 raft_consensus.cc:482] T b6be9a115fe14c098c29cad819849f95 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:28.033445 20372 raft_consensus.cc:3037] T b6be9a115fe14c098c29cad819849f95 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:28.034314 20372 raft_consensus.cc:504] T b6be9a115fe14c098c29cad819849f95 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:28.034626 20372 leader_election.cc:288] T b6be9a115fe14c098c29cad819849f95 P 99a0be6ef2144b7fb247ba9d3f051d66 [CANDIDATE]: Term 1 election: Requested vote from peers ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201), da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357)
I20240501 14:10:28.034914 19637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6be9a115fe14c098c29cad819849f95" candidate_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec8cde17a00645e7b38d5d29f7425367"
I20240501 14:10:28.035074 19637 raft_consensus.cc:3037] T b6be9a115fe14c098c29cad819849f95 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:28.035916 19637 raft_consensus.cc:2454] T b6be9a115fe14c098c29cad819849f95 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99a0be6ef2144b7fb247ba9d3f051d66 in term 1.
I20240501 14:10:28.036211 19441 leader_election.cc:302] T b6be9a115fe14c098c29cad819849f95 P 99a0be6ef2144b7fb247ba9d3f051d66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99a0be6ef2144b7fb247ba9d3f051d66, ec8cde17a00645e7b38d5d29f7425367; no voters:
I20240501 14:10:28.036216 19769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6be9a115fe14c098c29cad819849f95" candidate_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3c4e948ff747678f18456c134d7ed4" is_pre_election: true
I20240501 14:10:28.036232 19768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6be9a115fe14c098c29cad819849f95" candidate_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3c4e948ff747678f18456c134d7ed4"
I20240501 14:10:28.036381 19769 raft_consensus.cc:2454] T b6be9a115fe14c098c29cad819849f95 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99a0be6ef2144b7fb247ba9d3f051d66 in term 0.
I20240501 14:10:28.036577 20372 raft_consensus.cc:2781] T b6be9a115fe14c098c29cad819849f95 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:28.036784 20372 raft_consensus.cc:686] T b6be9a115fe14c098c29cad819849f95 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 LEADER]: Becoming Leader. State: Replica: 99a0be6ef2144b7fb247ba9d3f051d66, State: Running, Role: LEADER
I20240501 14:10:28.037107 20372 consensus_queue.cc:237] T b6be9a115fe14c098c29cad819849f95 P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:28.038663 19233 catalog_manager.cc:5484] T b6be9a115fe14c098c29cad819849f95 P 99a0be6ef2144b7fb247ba9d3f051d66 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130). New cstate: current_term: 1 leader_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } }
W20240501 14:10:28.151208 20616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:10:28.151428 20616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:10:28.154132 20616 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240501 14:10:28.235944 19637 raft_consensus.cc:1261] T ec2a46f8cfb3406082970e5fdb59b101 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:28.236701 20386 consensus_queue.cc:1035] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:28.239531 19375 raft_consensus.cc:1261] T ec2a46f8cfb3406082970e5fdb59b101 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:28.240229 19375 raft_consensus.cc:1261] T ffb4b6a2c19b497f8117fbbc9ea50082 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:28.240592 20378 consensus_queue.cc:1035] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:28.241098 19506 raft_consensus.cc:1261] T ffb4b6a2c19b497f8117fbbc9ea50082 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:28.241122 20386 consensus_queue.cc:1035] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:28.241472 20378 consensus_queue.cc:1035] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:28.246577 19638 raft_consensus.cc:1261] T b6be9a115fe14c098c29cad819849f95 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Refusing update from remote peer 99a0be6ef2144b7fb247ba9d3f051d66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:28.252125 19769 raft_consensus.cc:3037] T b6be9a115fe14c098c29cad819849f95 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:28.252451 20372 consensus_queue.cc:1035] T b6be9a115fe14c098c29cad819849f95 P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:28.253152 19769 raft_consensus.cc:1261] T b6be9a115fe14c098c29cad819849f95 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Refusing update from remote peer 99a0be6ef2144b7fb247ba9d3f051d66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:28.258306 20614 consensus_queue.cc:1035] T b6be9a115fe14c098c29cad819849f95 P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 113.836 ms
time per row: 0.0569178 ms
I20240501 14:10:28.308950 19226 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45968:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:10:28.314699 19618 tablet_service.cc:1467] Processing CreateTablet for tablet f69020bb6c244848a6b0a50fd2b4f3eb (DEFAULT_TABLE table=rebalance_test_table_4 [id=151dfad0ef40470ab9c5203c42400b60]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:28.314770 19355 tablet_service.cc:1467] Processing CreateTablet for tablet f69020bb6c244848a6b0a50fd2b4f3eb (DEFAULT_TABLE table=rebalance_test_table_4 [id=151dfad0ef40470ab9c5203c42400b60]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:28.315091 19618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f69020bb6c244848a6b0a50fd2b4f3eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:28.315126 19355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f69020bb6c244848a6b0a50fd2b4f3eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:28.315810 19487 tablet_service.cc:1467] Processing CreateTablet for tablet f69020bb6c244848a6b0a50fd2b4f3eb (DEFAULT_TABLE table=rebalance_test_table_4 [id=151dfad0ef40470ab9c5203c42400b60]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:28.315824 19617 tablet_service.cc:1467] Processing CreateTablet for tablet cf62635e9a994259b9e293e5eb425b55 (DEFAULT_TABLE table=rebalance_test_table_4 [id=151dfad0ef40470ab9c5203c42400b60]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:28.316067 19617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf62635e9a994259b9e293e5eb425b55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:28.316067 19487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f69020bb6c244848a6b0a50fd2b4f3eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:28.317369 20596 tablet_bootstrap.cc:492] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc: Bootstrap starting.
I20240501 14:10:28.317626 19485 tablet_service.cc:1467] Processing CreateTablet for tablet 814ba2247a314a4797aa0f9f624aaeb6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=151dfad0ef40470ab9c5203c42400b60]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:28.317879 19485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 814ba2247a314a4797aa0f9f624aaeb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:28.318238 20602 tablet_bootstrap.cc:492] T f69020bb6c244848a6b0a50fd2b4f3eb P 99a0be6ef2144b7fb247ba9d3f051d66: Bootstrap starting.
I20240501 14:10:28.318254 20596 tablet_bootstrap.cc:654] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:28.318320 19486 tablet_service.cc:1467] Processing CreateTablet for tablet cf62635e9a994259b9e293e5eb425b55 (DEFAULT_TABLE table=rebalance_test_table_4 [id=151dfad0ef40470ab9c5203c42400b60]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:28.318534 19486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf62635e9a994259b9e293e5eb425b55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:28.319156 20602 tablet_bootstrap.cc:654] T f69020bb6c244848a6b0a50fd2b4f3eb P 99a0be6ef2144b7fb247ba9d3f051d66: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:28.319229 20597 tablet_bootstrap.cc:492] T f69020bb6c244848a6b0a50fd2b4f3eb P ec8cde17a00645e7b38d5d29f7425367: Bootstrap starting.
I20240501 14:10:28.320184 20597 tablet_bootstrap.cc:654] T f69020bb6c244848a6b0a50fd2b4f3eb P ec8cde17a00645e7b38d5d29f7425367: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:28.320179 20602 tablet_bootstrap.cc:492] T f69020bb6c244848a6b0a50fd2b4f3eb P 99a0be6ef2144b7fb247ba9d3f051d66: No bootstrap required, opened a new log
I20240501 14:10:28.320259 20602 ts_tablet_manager.cc:1367] T f69020bb6c244848a6b0a50fd2b4f3eb P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:10:28.320760 20602 raft_consensus.cc:348] T f69020bb6c244848a6b0a50fd2b4f3eb P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:28.320884 20602 raft_consensus.cc:374] T f69020bb6c244848a6b0a50fd2b4f3eb P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:28.320922 20602 raft_consensus.cc:729] T f69020bb6c244848a6b0a50fd2b4f3eb P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a0be6ef2144b7fb247ba9d3f051d66, State: Initialized, Role: FOLLOWER
I20240501 14:10:28.321048 20602 consensus_queue.cc:260] T f69020bb6c244848a6b0a50fd2b4f3eb P 99a0be6ef2144b7fb247ba9d3f051d66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:28.321213 19749 tablet_service.cc:1467] Processing CreateTablet for tablet cf62635e9a994259b9e293e5eb425b55 (DEFAULT_TABLE table=rebalance_test_table_4 [id=151dfad0ef40470ab9c5203c42400b60]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:28.321375 20602 ts_tablet_manager.cc:1398] T f69020bb6c244848a6b0a50fd2b4f3eb P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:10:28.321213 19748 tablet_service.cc:1467] Processing CreateTablet for tablet 814ba2247a314a4797aa0f9f624aaeb6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=151dfad0ef40470ab9c5203c42400b60]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:28.321486 19749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf62635e9a994259b9e293e5eb425b55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:28.321552 20602 tablet_bootstrap.cc:492] T cf62635e9a994259b9e293e5eb425b55 P 99a0be6ef2144b7fb247ba9d3f051d66: Bootstrap starting.
I20240501 14:10:28.321601 19748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 814ba2247a314a4797aa0f9f624aaeb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:28.322502 20602 tablet_bootstrap.cc:654] T cf62635e9a994259b9e293e5eb425b55 P 99a0be6ef2144b7fb247ba9d3f051d66: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:28.323000 20600 tablet_bootstrap.cc:492] T cf62635e9a994259b9e293e5eb425b55 P da3c4e948ff747678f18456c134d7ed4: Bootstrap starting.
I20240501 14:10:28.323709 20602 tablet_bootstrap.cc:492] T cf62635e9a994259b9e293e5eb425b55 P 99a0be6ef2144b7fb247ba9d3f051d66: No bootstrap required, opened a new log
I20240501 14:10:28.323733 20600 tablet_bootstrap.cc:654] T cf62635e9a994259b9e293e5eb425b55 P da3c4e948ff747678f18456c134d7ed4: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:28.323779 20602 ts_tablet_manager.cc:1367] T cf62635e9a994259b9e293e5eb425b55 P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:10:28.324188 20602 raft_consensus.cc:348] T cf62635e9a994259b9e293e5eb425b55 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:28.324282 20602 raft_consensus.cc:374] T cf62635e9a994259b9e293e5eb425b55 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:28.324318 20602 raft_consensus.cc:729] T cf62635e9a994259b9e293e5eb425b55 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a0be6ef2144b7fb247ba9d3f051d66, State: Initialized, Role: FOLLOWER
I20240501 14:10:28.324466 20600 tablet_bootstrap.cc:492] T cf62635e9a994259b9e293e5eb425b55 P da3c4e948ff747678f18456c134d7ed4: No bootstrap required, opened a new log
I20240501 14:10:28.324468 20602 consensus_queue.cc:260] T cf62635e9a994259b9e293e5eb425b55 P 99a0be6ef2144b7fb247ba9d3f051d66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:28.324550 20600 ts_tablet_manager.cc:1367] T cf62635e9a994259b9e293e5eb425b55 P da3c4e948ff747678f18456c134d7ed4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:10:28.324771 20602 ts_tablet_manager.cc:1398] T cf62635e9a994259b9e293e5eb425b55 P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:10:28.324887 20600 raft_consensus.cc:348] T cf62635e9a994259b9e293e5eb425b55 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:28.324959 20600 raft_consensus.cc:374] T cf62635e9a994259b9e293e5eb425b55 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:28.324977 20600 raft_consensus.cc:729] T cf62635e9a994259b9e293e5eb425b55 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3c4e948ff747678f18456c134d7ed4, State: Initialized, Role: FOLLOWER
I20240501 14:10:28.325078 20600 consensus_queue.cc:260] T cf62635e9a994259b9e293e5eb425b55 P da3c4e948ff747678f18456c134d7ed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:28.325318 20600 ts_tablet_manager.cc:1398] T cf62635e9a994259b9e293e5eb425b55 P da3c4e948ff747678f18456c134d7ed4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:10:28.325538 20600 tablet_bootstrap.cc:492] T 814ba2247a314a4797aa0f9f624aaeb6 P da3c4e948ff747678f18456c134d7ed4: Bootstrap starting.
I20240501 14:10:28.326295 20600 tablet_bootstrap.cc:654] T 814ba2247a314a4797aa0f9f624aaeb6 P da3c4e948ff747678f18456c134d7ed4: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:28.326701 19355 tablet_service.cc:1467] Processing CreateTablet for tablet 814ba2247a314a4797aa0f9f624aaeb6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=151dfad0ef40470ab9c5203c42400b60]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:28.326988 19355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 814ba2247a314a4797aa0f9f624aaeb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:28.327842 20597 tablet_bootstrap.cc:492] T f69020bb6c244848a6b0a50fd2b4f3eb P ec8cde17a00645e7b38d5d29f7425367: No bootstrap required, opened a new log
I20240501 14:10:28.327944 20597 ts_tablet_manager.cc:1367] T f69020bb6c244848a6b0a50fd2b4f3eb P ec8cde17a00645e7b38d5d29f7425367: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240501 14:10:28.328446 20597 raft_consensus.cc:348] T f69020bb6c244848a6b0a50fd2b4f3eb P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:28.328572 20597 raft_consensus.cc:374] T f69020bb6c244848a6b0a50fd2b4f3eb P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:28.328609 20597 raft_consensus.cc:729] T f69020bb6c244848a6b0a50fd2b4f3eb P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec8cde17a00645e7b38d5d29f7425367, State: Initialized, Role: FOLLOWER
I20240501 14:10:28.328725 20597 consensus_queue.cc:260] T f69020bb6c244848a6b0a50fd2b4f3eb P ec8cde17a00645e7b38d5d29f7425367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:28.329039 20597 ts_tablet_manager.cc:1398] T f69020bb6c244848a6b0a50fd2b4f3eb P ec8cde17a00645e7b38d5d29f7425367: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:28.329224 20597 tablet_bootstrap.cc:492] T cf62635e9a994259b9e293e5eb425b55 P ec8cde17a00645e7b38d5d29f7425367: Bootstrap starting.
I20240501 14:10:28.330070 20597 tablet_bootstrap.cc:654] T cf62635e9a994259b9e293e5eb425b55 P ec8cde17a00645e7b38d5d29f7425367: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:28.330957 20597 tablet_bootstrap.cc:492] T cf62635e9a994259b9e293e5eb425b55 P ec8cde17a00645e7b38d5d29f7425367: No bootstrap required, opened a new log
I20240501 14:10:28.331099 20597 ts_tablet_manager.cc:1367] T cf62635e9a994259b9e293e5eb425b55 P ec8cde17a00645e7b38d5d29f7425367: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:10:28.331480 20602 tablet_bootstrap.cc:492] T 814ba2247a314a4797aa0f9f624aaeb6 P 99a0be6ef2144b7fb247ba9d3f051d66: Bootstrap starting.
I20240501 14:10:28.331584 20597 raft_consensus.cc:348] T cf62635e9a994259b9e293e5eb425b55 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:28.331755 20597 raft_consensus.cc:374] T cf62635e9a994259b9e293e5eb425b55 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:28.332032 20596 tablet_bootstrap.cc:492] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc: No bootstrap required, opened a new log
I20240501 14:10:28.332131 20596 ts_tablet_manager.cc:1367] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.000s
I20240501 14:10:28.332602 20596 raft_consensus.cc:348] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:28.331827 20597 raft_consensus.cc:729] T cf62635e9a994259b9e293e5eb425b55 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec8cde17a00645e7b38d5d29f7425367, State: Initialized, Role: FOLLOWER
I20240501 14:10:28.332717 20596 raft_consensus.cc:374] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:28.332762 20596 raft_consensus.cc:729] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1e6bc2d07b4ce3b622a66f14c93bdc, State: Initialized, Role: FOLLOWER
I20240501 14:10:28.332813 20597 consensus_queue.cc:260] T cf62635e9a994259b9e293e5eb425b55 P ec8cde17a00645e7b38d5d29f7425367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:28.332880 20596 consensus_queue.cc:260] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:28.332914 20602 tablet_bootstrap.cc:654] T 814ba2247a314a4797aa0f9f624aaeb6 P 99a0be6ef2144b7fb247ba9d3f051d66: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:28.333117 20597 ts_tablet_manager.cc:1398] T cf62635e9a994259b9e293e5eb425b55 P ec8cde17a00645e7b38d5d29f7425367: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:10:28.333137 20596 ts_tablet_manager.cc:1398] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:28.333288 20596 tablet_bootstrap.cc:492] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Bootstrap starting.
I20240501 14:10:28.333943 20602 tablet_bootstrap.cc:492] T 814ba2247a314a4797aa0f9f624aaeb6 P 99a0be6ef2144b7fb247ba9d3f051d66: No bootstrap required, opened a new log
I20240501 14:10:28.334013 20602 ts_tablet_manager.cc:1367] T 814ba2247a314a4797aa0f9f624aaeb6 P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:10:28.334250 20596 tablet_bootstrap.cc:654] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:28.334477 20602 raft_consensus.cc:348] T 814ba2247a314a4797aa0f9f624aaeb6 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:28.334583 20602 raft_consensus.cc:374] T 814ba2247a314a4797aa0f9f624aaeb6 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:28.334617 20602 raft_consensus.cc:729] T 814ba2247a314a4797aa0f9f624aaeb6 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a0be6ef2144b7fb247ba9d3f051d66, State: Initialized, Role: FOLLOWER
I20240501 14:10:28.334733 20602 consensus_queue.cc:260] T 814ba2247a314a4797aa0f9f624aaeb6 P 99a0be6ef2144b7fb247ba9d3f051d66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:28.334985 20602 ts_tablet_manager.cc:1398] T 814ba2247a314a4797aa0f9f624aaeb6 P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:10:28.336088 20596 tablet_bootstrap.cc:492] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc: No bootstrap required, opened a new log
I20240501 14:10:28.336097 20600 tablet_bootstrap.cc:492] T 814ba2247a314a4797aa0f9f624aaeb6 P da3c4e948ff747678f18456c134d7ed4: No bootstrap required, opened a new log
I20240501 14:10:28.336171 20600 ts_tablet_manager.cc:1367] T 814ba2247a314a4797aa0f9f624aaeb6 P da3c4e948ff747678f18456c134d7ed4: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20240501 14:10:28.336171 20596 ts_tablet_manager.cc:1367] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:10:28.336570 20596 raft_consensus.cc:348] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:28.336570 20600 raft_consensus.cc:348] T 814ba2247a314a4797aa0f9f624aaeb6 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:28.336669 20600 raft_consensus.cc:374] T 814ba2247a314a4797aa0f9f624aaeb6 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:28.336668 20596 raft_consensus.cc:374] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:28.336699 20600 raft_consensus.cc:729] T 814ba2247a314a4797aa0f9f624aaeb6 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3c4e948ff747678f18456c134d7ed4, State: Initialized, Role: FOLLOWER
I20240501 14:10:28.336699 20596 raft_consensus.cc:729] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1e6bc2d07b4ce3b622a66f14c93bdc, State: Initialized, Role: FOLLOWER
I20240501 14:10:28.336800 20596 consensus_queue.cc:260] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:28.336802 20600 consensus_queue.cc:260] T 814ba2247a314a4797aa0f9f624aaeb6 P da3c4e948ff747678f18456c134d7ed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:28.337052 20596 ts_tablet_manager.cc:1398] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:28.337086 20600 ts_tablet_manager.cc:1398] T 814ba2247a314a4797aa0f9f624aaeb6 P da3c4e948ff747678f18456c134d7ed4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:28.340322 20610 raft_consensus.cc:482] T cf62635e9a994259b9e293e5eb425b55 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:28.340405 20610 raft_consensus.cc:504] T cf62635e9a994259b9e293e5eb425b55 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:28.340725 20610 leader_election.cc:288] T cf62635e9a994259b9e293e5eb425b55 P da3c4e948ff747678f18456c134d7ed4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201), 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829)
I20240501 14:10:28.341001 19637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf62635e9a994259b9e293e5eb425b55" candidate_uuid: "da3c4e948ff747678f18456c134d7ed4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec8cde17a00645e7b38d5d29f7425367" is_pre_election: true
I20240501 14:10:28.341063 19506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf62635e9a994259b9e293e5eb425b55" candidate_uuid: "da3c4e948ff747678f18456c134d7ed4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" is_pre_election: true
I20240501 14:10:28.341115 19637 raft_consensus.cc:2454] T cf62635e9a994259b9e293e5eb425b55 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da3c4e948ff747678f18456c134d7ed4 in term 0.
I20240501 14:10:28.341173 19506 raft_consensus.cc:2454] T cf62635e9a994259b9e293e5eb425b55 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da3c4e948ff747678f18456c134d7ed4 in term 0.
I20240501 14:10:28.341441 19703 leader_election.cc:302] T cf62635e9a994259b9e293e5eb425b55 P da3c4e948ff747678f18456c134d7ed4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da3c4e948ff747678f18456c134d7ed4, ec8cde17a00645e7b38d5d29f7425367; no voters:
I20240501 14:10:28.341696 20610 raft_consensus.cc:2781] T cf62635e9a994259b9e293e5eb425b55 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:28.341744 20610 raft_consensus.cc:482] T cf62635e9a994259b9e293e5eb425b55 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:28.341785 20610 raft_consensus.cc:3037] T cf62635e9a994259b9e293e5eb425b55 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:28.342660 20610 raft_consensus.cc:504] T cf62635e9a994259b9e293e5eb425b55 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:28.342926 20610 leader_election.cc:288] T cf62635e9a994259b9e293e5eb425b55 P da3c4e948ff747678f18456c134d7ed4 [CANDIDATE]: Term 1 election: Requested vote from peers ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201), 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829)
I20240501 14:10:28.343255 19637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf62635e9a994259b9e293e5eb425b55" candidate_uuid: "da3c4e948ff747678f18456c134d7ed4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec8cde17a00645e7b38d5d29f7425367"
I20240501 14:10:28.343304 19506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf62635e9a994259b9e293e5eb425b55" candidate_uuid: "da3c4e948ff747678f18456c134d7ed4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a0be6ef2144b7fb247ba9d3f051d66"
I20240501 14:10:28.343355 19637 raft_consensus.cc:3037] T cf62635e9a994259b9e293e5eb425b55 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:28.343400 19506 raft_consensus.cc:3037] T cf62635e9a994259b9e293e5eb425b55 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:28.344246 19637 raft_consensus.cc:2454] T cf62635e9a994259b9e293e5eb425b55 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da3c4e948ff747678f18456c134d7ed4 in term 1.
I20240501 14:10:28.344312 19506 raft_consensus.cc:2454] T cf62635e9a994259b9e293e5eb425b55 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da3c4e948ff747678f18456c134d7ed4 in term 1.
I20240501 14:10:28.344508 19703 leader_election.cc:302] T cf62635e9a994259b9e293e5eb425b55 P da3c4e948ff747678f18456c134d7ed4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da3c4e948ff747678f18456c134d7ed4, ec8cde17a00645e7b38d5d29f7425367; no voters:
I20240501 14:10:28.344655 20610 raft_consensus.cc:2781] T cf62635e9a994259b9e293e5eb425b55 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:28.344733 20610 raft_consensus.cc:686] T cf62635e9a994259b9e293e5eb425b55 P da3c4e948ff747678f18456c134d7ed4 [term 1 LEADER]: Becoming Leader. State: Replica: da3c4e948ff747678f18456c134d7ed4, State: Running, Role: LEADER
I20240501 14:10:28.344894 20610 consensus_queue.cc:237] T cf62635e9a994259b9e293e5eb425b55 P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:28.346172 19226 catalog_manager.cc:5484] T cf62635e9a994259b9e293e5eb425b55 P da3c4e948ff747678f18456c134d7ed4 reported cstate change: term changed from 0 to 1, leader changed from <none> to da3c4e948ff747678f18456c134d7ed4 (127.7.39.132). New cstate: current_term: 1 leader_uuid: "da3c4e948ff747678f18456c134d7ed4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } }
I20240501 14:10:28.359516 20499 raft_consensus.cc:482] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:28.359637 20499 raft_consensus.cc:504] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:28.360036 20499 leader_election.cc:288] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357), 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829)
I20240501 14:10:28.360392 19506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "814ba2247a314a4797aa0f9f624aaeb6" candidate_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" is_pre_election: true
I20240501 14:10:28.360395 19769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "814ba2247a314a4797aa0f9f624aaeb6" candidate_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3c4e948ff747678f18456c134d7ed4" is_pre_election: true
I20240501 14:10:28.360524 19506 raft_consensus.cc:2454] T 814ba2247a314a4797aa0f9f624aaeb6 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb1e6bc2d07b4ce3b622a66f14c93bdc in term 0.
I20240501 14:10:28.360524 19769 raft_consensus.cc:2454] T 814ba2247a314a4797aa0f9f624aaeb6 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb1e6bc2d07b4ce3b622a66f14c93bdc in term 0.
I20240501 14:10:28.360806 19311 leader_election.cc:302] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da3c4e948ff747678f18456c134d7ed4, fb1e6bc2d07b4ce3b622a66f14c93bdc; no voters:
I20240501 14:10:28.360970 20499 raft_consensus.cc:2781] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:28.361033 20499 raft_consensus.cc:482] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:28.361093 20499 raft_consensus.cc:3037] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:28.361866 20499 raft_consensus.cc:504] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:28.362179 20499 leader_election.cc:288] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc [CANDIDATE]: Term 1 election: Requested vote from peers da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357), 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829)
I20240501 14:10:28.362509 19506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "814ba2247a314a4797aa0f9f624aaeb6" candidate_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a0be6ef2144b7fb247ba9d3f051d66"
I20240501 14:10:28.362509 19769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "814ba2247a314a4797aa0f9f624aaeb6" candidate_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3c4e948ff747678f18456c134d7ed4"
I20240501 14:10:28.362653 19506 raft_consensus.cc:3037] T 814ba2247a314a4797aa0f9f624aaeb6 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:28.362653 19769 raft_consensus.cc:3037] T 814ba2247a314a4797aa0f9f624aaeb6 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:28.363626 19506 raft_consensus.cc:2454] T 814ba2247a314a4797aa0f9f624aaeb6 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb1e6bc2d07b4ce3b622a66f14c93bdc in term 1.
I20240501 14:10:28.363631 19769 raft_consensus.cc:2454] T 814ba2247a314a4797aa0f9f624aaeb6 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb1e6bc2d07b4ce3b622a66f14c93bdc in term 1.
I20240501 14:10:28.363945 19310 leader_election.cc:302] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99a0be6ef2144b7fb247ba9d3f051d66, fb1e6bc2d07b4ce3b622a66f14c93bdc; no voters:
I20240501 14:10:28.364157 20499 raft_consensus.cc:2781] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:28.364239 20499 raft_consensus.cc:686] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Becoming Leader. State: Replica: fb1e6bc2d07b4ce3b622a66f14c93bdc, State: Running, Role: LEADER
I20240501 14:10:28.364380 20499 consensus_queue.cc:237] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:28.366103 19226 catalog_manager.cc:5484] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc reported cstate change: term changed from 0 to 1, leader changed from <none> to fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129). New cstate: current_term: 1 leader_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } }
I20240501 14:10:28.528070 20499 raft_consensus.cc:482] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:28.528224 20499 raft_consensus.cc:504] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:28.528733 20499 leader_election.cc:288] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201), 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829)
I20240501 14:10:28.529544 19506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69020bb6c244848a6b0a50fd2b4f3eb" candidate_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" is_pre_election: true
I20240501 14:10:28.529726 19506 raft_consensus.cc:2454] T f69020bb6c244848a6b0a50fd2b4f3eb P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb1e6bc2d07b4ce3b622a66f14c93bdc in term 0.
I20240501 14:10:28.530050 19310 leader_election.cc:302] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99a0be6ef2144b7fb247ba9d3f051d66, fb1e6bc2d07b4ce3b622a66f14c93bdc; no voters:
I20240501 14:10:28.530212 20499 raft_consensus.cc:2781] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:28.530265 20499 raft_consensus.cc:482] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:28.530298 20499 raft_consensus.cc:3037] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:28.531255 20499 raft_consensus.cc:504] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:28.531586 20499 leader_election.cc:288] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc [CANDIDATE]: Term 1 election: Requested vote from peers ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201), 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829)
I20240501 14:10:28.531930 19506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69020bb6c244848a6b0a50fd2b4f3eb" candidate_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a0be6ef2144b7fb247ba9d3f051d66"
I20240501 14:10:28.532048 19506 raft_consensus.cc:3037] T f69020bb6c244848a6b0a50fd2b4f3eb P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:28.532662 19637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69020bb6c244848a6b0a50fd2b4f3eb" candidate_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec8cde17a00645e7b38d5d29f7425367" is_pre_election: true
I20240501 14:10:28.532753 19638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69020bb6c244848a6b0a50fd2b4f3eb" candidate_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec8cde17a00645e7b38d5d29f7425367"
I20240501 14:10:28.532791 19637 raft_consensus.cc:2454] T f69020bb6c244848a6b0a50fd2b4f3eb P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb1e6bc2d07b4ce3b622a66f14c93bdc in term 0.
I20240501 14:10:28.532819 19506 raft_consensus.cc:2454] T f69020bb6c244848a6b0a50fd2b4f3eb P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb1e6bc2d07b4ce3b622a66f14c93bdc in term 1.
I20240501 14:10:28.533255 19310 leader_election.cc:302] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99a0be6ef2144b7fb247ba9d3f051d66, fb1e6bc2d07b4ce3b622a66f14c93bdc; no voters:
I20240501 14:10:28.533421 20499 raft_consensus.cc:2781] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:28.533510 20499 raft_consensus.cc:686] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Becoming Leader. State: Replica: fb1e6bc2d07b4ce3b622a66f14c93bdc, State: Running, Role: LEADER
I20240501 14:10:28.533664 20499 consensus_queue.cc:237] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:28.535290 19226 catalog_manager.cc:5484] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc reported cstate change: term changed from 0 to 1, leader changed from <none> to fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129). New cstate: current_term: 1 leader_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } }
W20240501 14:10:28.664885 20669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:10:28.665150 20669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:10:28.668118 20669 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240501 14:10:28.745221 20499 consensus_queue.cc:1035] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:28.753024 19507 raft_consensus.cc:1261] T cf62635e9a994259b9e293e5eb425b55 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:28.753401 19638 raft_consensus.cc:1261] T cf62635e9a994259b9e293e5eb425b55 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:28.754714 20610 consensus_queue.cc:1035] T cf62635e9a994259b9e293e5eb425b55 P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:28.755017 20610 consensus_queue.cc:1035] T cf62635e9a994259b9e293e5eb425b55 P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:28.755897 20499 consensus_queue.cc:1035] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:28.758344 19638 raft_consensus.cc:3037] T f69020bb6c244848a6b0a50fd2b4f3eb P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:28.759451 19638 raft_consensus.cc:1261] T f69020bb6c244848a6b0a50fd2b4f3eb P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:28.759770 20511 consensus_queue.cc:1035] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:28.759852 19507 raft_consensus.cc:1261] T f69020bb6c244848a6b0a50fd2b4f3eb P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:28.760284 20511 consensus_queue.cc:1035] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 108.815 ms
time per row: 0.0544073 ms
I20240501 14:10:28.817412 19231 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45968:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:10:28.822760 19617 tablet_service.cc:1467] Processing CreateTablet for tablet 12ffa74f7a5b472aac57253693273cad (DEFAULT_TABLE table=rebalance_test_table_5 [id=4d7621338643426d90fe03eb71654cb8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:28.822760 19355 tablet_service.cc:1467] Processing CreateTablet for tablet 12ffa74f7a5b472aac57253693273cad (DEFAULT_TABLE table=rebalance_test_table_5 [id=4d7621338643426d90fe03eb71654cb8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:28.823143 19617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12ffa74f7a5b472aac57253693273cad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:28.823143 19355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12ffa74f7a5b472aac57253693273cad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:28.825285 20596 tablet_bootstrap.cc:492] T 12ffa74f7a5b472aac57253693273cad P fb1e6bc2d07b4ce3b622a66f14c93bdc: Bootstrap starting.
I20240501 14:10:28.825829 19617 tablet_service.cc:1467] Processing CreateTablet for tablet bb291a0c743c4e638f79da36cc77faa3 (DEFAULT_TABLE table=rebalance_test_table_5 [id=4d7621338643426d90fe03eb71654cb8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:28.825850 19618 tablet_service.cc:1467] Processing CreateTablet for tablet 98af8b06b9324e3fa7cdb898b7b9fab9 (DEFAULT_TABLE table=rebalance_test_table_5 [id=4d7621338643426d90fe03eb71654cb8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:28.826093 19617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb291a0c743c4e638f79da36cc77faa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:28.826120 19485 tablet_service.cc:1467] Processing CreateTablet for tablet 98af8b06b9324e3fa7cdb898b7b9fab9 (DEFAULT_TABLE table=rebalance_test_table_5 [id=4d7621338643426d90fe03eb71654cb8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:28.826197 20596 tablet_bootstrap.cc:654] T 12ffa74f7a5b472aac57253693273cad P fb1e6bc2d07b4ce3b622a66f14c93bdc: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:28.826234 19618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98af8b06b9324e3fa7cdb898b7b9fab9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:28.826380 19485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98af8b06b9324e3fa7cdb898b7b9fab9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:28.826790 20597 tablet_bootstrap.cc:492] T 12ffa74f7a5b472aac57253693273cad P ec8cde17a00645e7b38d5d29f7425367: Bootstrap starting.
I20240501 14:10:28.827744 20597 tablet_bootstrap.cc:654] T 12ffa74f7a5b472aac57253693273cad P ec8cde17a00645e7b38d5d29f7425367: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:28.828452 20602 tablet_bootstrap.cc:492] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 99a0be6ef2144b7fb247ba9d3f051d66: Bootstrap starting.
I20240501 14:10:28.828847 20597 tablet_bootstrap.cc:492] T 12ffa74f7a5b472aac57253693273cad P ec8cde17a00645e7b38d5d29f7425367: No bootstrap required, opened a new log
I20240501 14:10:28.828939 20597 ts_tablet_manager.cc:1367] T 12ffa74f7a5b472aac57253693273cad P ec8cde17a00645e7b38d5d29f7425367: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:10:28.829267 19748 tablet_service.cc:1467] Processing CreateTablet for tablet 12ffa74f7a5b472aac57253693273cad (DEFAULT_TABLE table=rebalance_test_table_5 [id=4d7621338643426d90fe03eb71654cb8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:28.829408 20602 tablet_bootstrap.cc:654] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 99a0be6ef2144b7fb247ba9d3f051d66: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:28.829324 20597 raft_consensus.cc:348] T 12ffa74f7a5b472aac57253693273cad P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:28.829449 20597 raft_consensus.cc:374] T 12ffa74f7a5b472aac57253693273cad P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:28.829484 20597 raft_consensus.cc:729] T 12ffa74f7a5b472aac57253693273cad P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec8cde17a00645e7b38d5d29f7425367, State: Initialized, Role: FOLLOWER
I20240501 14:10:28.829582 19748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12ffa74f7a5b472aac57253693273cad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:28.829599 20597 consensus_queue.cc:260] T 12ffa74f7a5b472aac57253693273cad P ec8cde17a00645e7b38d5d29f7425367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:28.829869 20597 ts_tablet_manager.cc:1398] T 12ffa74f7a5b472aac57253693273cad P ec8cde17a00645e7b38d5d29f7425367: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:28.830034 20597 tablet_bootstrap.cc:492] T 98af8b06b9324e3fa7cdb898b7b9fab9 P ec8cde17a00645e7b38d5d29f7425367: Bootstrap starting.
I20240501 14:10:28.830859 20597 tablet_bootstrap.cc:654] T 98af8b06b9324e3fa7cdb898b7b9fab9 P ec8cde17a00645e7b38d5d29f7425367: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:28.831014 20602 tablet_bootstrap.cc:492] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 99a0be6ef2144b7fb247ba9d3f051d66: No bootstrap required, opened a new log
I20240501 14:10:28.831137 20602 ts_tablet_manager.cc:1367] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:10:28.831612 20602 raft_consensus.cc:348] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:28.831712 20602 raft_consensus.cc:374] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:28.831760 20602 raft_consensus.cc:729] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a0be6ef2144b7fb247ba9d3f051d66, State: Initialized, Role: FOLLOWER
I20240501 14:10:28.831884 20602 consensus_queue.cc:260] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 99a0be6ef2144b7fb247ba9d3f051d66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:28.831952 19749 tablet_service.cc:1467] Processing CreateTablet for tablet bb291a0c743c4e638f79da36cc77faa3 (DEFAULT_TABLE table=rebalance_test_table_5 [id=4d7621338643426d90fe03eb71654cb8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:28.832031 20597 tablet_bootstrap.cc:492] T 98af8b06b9324e3fa7cdb898b7b9fab9 P ec8cde17a00645e7b38d5d29f7425367: No bootstrap required, opened a new log
I20240501 14:10:28.832132 20597 ts_tablet_manager.cc:1367] T 98af8b06b9324e3fa7cdb898b7b9fab9 P ec8cde17a00645e7b38d5d29f7425367: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:10:28.832211 20602 ts_tablet_manager.cc:1398] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:10:28.832237 19749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb291a0c743c4e638f79da36cc77faa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:28.832585 20597 raft_consensus.cc:348] T 98af8b06b9324e3fa7cdb898b7b9fab9 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:28.832684 20597 raft_consensus.cc:374] T 98af8b06b9324e3fa7cdb898b7b9fab9 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:28.832712 20597 raft_consensus.cc:729] T 98af8b06b9324e3fa7cdb898b7b9fab9 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec8cde17a00645e7b38d5d29f7425367, State: Initialized, Role: FOLLOWER
I20240501 14:10:28.832815 20597 consensus_queue.cc:260] T 98af8b06b9324e3fa7cdb898b7b9fab9 P ec8cde17a00645e7b38d5d29f7425367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:28.833184 20597 ts_tablet_manager.cc:1398] T 98af8b06b9324e3fa7cdb898b7b9fab9 P ec8cde17a00645e7b38d5d29f7425367: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:28.833362 20597 tablet_bootstrap.cc:492] T bb291a0c743c4e638f79da36cc77faa3 P ec8cde17a00645e7b38d5d29f7425367: Bootstrap starting.
I20240501 14:10:28.834082 20600 tablet_bootstrap.cc:492] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4: Bootstrap starting.
I20240501 14:10:28.834156 20597 tablet_bootstrap.cc:654] T bb291a0c743c4e638f79da36cc77faa3 P ec8cde17a00645e7b38d5d29f7425367: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:28.834841 20600 tablet_bootstrap.cc:654] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:28.835441 20597 tablet_bootstrap.cc:492] T bb291a0c743c4e638f79da36cc77faa3 P ec8cde17a00645e7b38d5d29f7425367: No bootstrap required, opened a new log
I20240501 14:10:28.835523 20597 ts_tablet_manager.cc:1367] T bb291a0c743c4e638f79da36cc77faa3 P ec8cde17a00645e7b38d5d29f7425367: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:10:28.835836 20600 tablet_bootstrap.cc:492] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4: No bootstrap required, opened a new log
I20240501 14:10:28.835908 20600 ts_tablet_manager.cc:1367] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:10:28.835875 19355 tablet_service.cc:1467] Processing CreateTablet for tablet bb291a0c743c4e638f79da36cc77faa3 (DEFAULT_TABLE table=rebalance_test_table_5 [id=4d7621338643426d90fe03eb71654cb8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:28.835916 20597 raft_consensus.cc:348] T bb291a0c743c4e638f79da36cc77faa3 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:28.836074 20597 raft_consensus.cc:374] T bb291a0c743c4e638f79da36cc77faa3 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:28.836122 20597 raft_consensus.cc:729] T bb291a0c743c4e638f79da36cc77faa3 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec8cde17a00645e7b38d5d29f7425367, State: Initialized, Role: FOLLOWER
I20240501 14:10:28.836145 19355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb291a0c743c4e638f79da36cc77faa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:28.836256 20597 consensus_queue.cc:260] T bb291a0c743c4e638f79da36cc77faa3 P ec8cde17a00645e7b38d5d29f7425367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:28.836323 20600 raft_consensus.cc:348] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:28.836431 20600 raft_consensus.cc:374] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:28.836462 20600 raft_consensus.cc:729] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3c4e948ff747678f18456c134d7ed4, State: Initialized, Role: FOLLOWER
I20240501 14:10:28.836584 20600 consensus_queue.cc:260] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:28.836642 20597 ts_tablet_manager.cc:1398] T bb291a0c743c4e638f79da36cc77faa3 P ec8cde17a00645e7b38d5d29f7425367: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:28.836843 20600 ts_tablet_manager.cc:1398] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:10:28.836992 20600 tablet_bootstrap.cc:492] T bb291a0c743c4e638f79da36cc77faa3 P da3c4e948ff747678f18456c134d7ed4: Bootstrap starting.
I20240501 14:10:28.837836 20600 tablet_bootstrap.cc:654] T bb291a0c743c4e638f79da36cc77faa3 P da3c4e948ff747678f18456c134d7ed4: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:28.838534 19356 tablet_service.cc:1467] Processing CreateTablet for tablet 98af8b06b9324e3fa7cdb898b7b9fab9 (DEFAULT_TABLE table=rebalance_test_table_5 [id=4d7621338643426d90fe03eb71654cb8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:28.838740 19356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98af8b06b9324e3fa7cdb898b7b9fab9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:28.839839 20596 tablet_bootstrap.cc:492] T 12ffa74f7a5b472aac57253693273cad P fb1e6bc2d07b4ce3b622a66f14c93bdc: No bootstrap required, opened a new log
I20240501 14:10:28.839926 20596 ts_tablet_manager.cc:1367] T 12ffa74f7a5b472aac57253693273cad P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240501 14:10:28.840258 20596 raft_consensus.cc:348] T 12ffa74f7a5b472aac57253693273cad P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:28.840342 20596 raft_consensus.cc:374] T 12ffa74f7a5b472aac57253693273cad P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:28.840363 20596 raft_consensus.cc:729] T 12ffa74f7a5b472aac57253693273cad P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1e6bc2d07b4ce3b622a66f14c93bdc, State: Initialized, Role: FOLLOWER
I20240501 14:10:28.840451 20596 consensus_queue.cc:260] T 12ffa74f7a5b472aac57253693273cad P fb1e6bc2d07b4ce3b622a66f14c93bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:28.840687 20596 ts_tablet_manager.cc:1398] T 12ffa74f7a5b472aac57253693273cad P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:10:28.840900 20596 tablet_bootstrap.cc:492] T bb291a0c743c4e638f79da36cc77faa3 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Bootstrap starting.
I20240501 14:10:28.841756 20600 tablet_bootstrap.cc:492] T bb291a0c743c4e638f79da36cc77faa3 P da3c4e948ff747678f18456c134d7ed4: No bootstrap required, opened a new log
I20240501 14:10:28.841800 20596 tablet_bootstrap.cc:654] T bb291a0c743c4e638f79da36cc77faa3 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:28.841840 20600 ts_tablet_manager.cc:1367] T bb291a0c743c4e638f79da36cc77faa3 P da3c4e948ff747678f18456c134d7ed4: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:10:28.842437 20600 raft_consensus.cc:348] T bb291a0c743c4e638f79da36cc77faa3 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:28.842545 20600 raft_consensus.cc:374] T bb291a0c743c4e638f79da36cc77faa3 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:28.842581 20600 raft_consensus.cc:729] T bb291a0c743c4e638f79da36cc77faa3 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3c4e948ff747678f18456c134d7ed4, State: Initialized, Role: FOLLOWER
I20240501 14:10:28.842722 20600 consensus_queue.cc:260] T bb291a0c743c4e638f79da36cc77faa3 P da3c4e948ff747678f18456c134d7ed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:28.843035 20600 ts_tablet_manager.cc:1398] T bb291a0c743c4e638f79da36cc77faa3 P da3c4e948ff747678f18456c134d7ed4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:10:28.843308 20596 tablet_bootstrap.cc:492] T bb291a0c743c4e638f79da36cc77faa3 P fb1e6bc2d07b4ce3b622a66f14c93bdc: No bootstrap required, opened a new log
I20240501 14:10:28.843384 20596 ts_tablet_manager.cc:1367] T bb291a0c743c4e638f79da36cc77faa3 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:10:28.843962 20596 raft_consensus.cc:348] T bb291a0c743c4e638f79da36cc77faa3 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:28.844091 20596 raft_consensus.cc:374] T bb291a0c743c4e638f79da36cc77faa3 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:28.844139 20596 raft_consensus.cc:729] T bb291a0c743c4e638f79da36cc77faa3 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1e6bc2d07b4ce3b622a66f14c93bdc, State: Initialized, Role: FOLLOWER
I20240501 14:10:28.844285 20596 consensus_queue.cc:260] T bb291a0c743c4e638f79da36cc77faa3 P fb1e6bc2d07b4ce3b622a66f14c93bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:28.844570 20596 ts_tablet_manager.cc:1398] T bb291a0c743c4e638f79da36cc77faa3 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:28.844693 20596 tablet_bootstrap.cc:492] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Bootstrap starting.
I20240501 14:10:28.845655 20596 tablet_bootstrap.cc:654] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:28.846542 20596 tablet_bootstrap.cc:492] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc: No bootstrap required, opened a new log
I20240501 14:10:28.846608 20596 ts_tablet_manager.cc:1367] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:10:28.846896 20596 raft_consensus.cc:348] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:28.846971 20596 raft_consensus.cc:374] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:28.846992 20596 raft_consensus.cc:729] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1e6bc2d07b4ce3b622a66f14c93bdc, State: Initialized, Role: FOLLOWER
I20240501 14:10:28.847142 20596 consensus_queue.cc:260] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:28.847383 20596 ts_tablet_manager.cc:1398] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:10:28.863191 20499 raft_consensus.cc:482] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:28.863303 20499 raft_consensus.cc:504] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:28.863655 20499 leader_election.cc:288] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829), ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201)
I20240501 14:10:28.864003 19505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98af8b06b9324e3fa7cdb898b7b9fab9" candidate_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" is_pre_election: true
I20240501 14:10:28.864171 19505 raft_consensus.cc:2454] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb1e6bc2d07b4ce3b622a66f14c93bdc in term 0.
I20240501 14:10:28.864138 19638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98af8b06b9324e3fa7cdb898b7b9fab9" candidate_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec8cde17a00645e7b38d5d29f7425367" is_pre_election: true
I20240501 14:10:28.864295 19638 raft_consensus.cc:2454] T 98af8b06b9324e3fa7cdb898b7b9fab9 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb1e6bc2d07b4ce3b622a66f14c93bdc in term 0.
I20240501 14:10:28.864468 19310 leader_election.cc:302] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99a0be6ef2144b7fb247ba9d3f051d66, fb1e6bc2d07b4ce3b622a66f14c93bdc; no voters:
I20240501 14:10:28.864646 20499 raft_consensus.cc:2781] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:28.864706 20499 raft_consensus.cc:482] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:28.864756 20499 raft_consensus.cc:3037] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:28.865566 20499 raft_consensus.cc:504] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:28.865911 20499 leader_election.cc:288] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [CANDIDATE]: Term 1 election: Requested vote from peers 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829), ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201)
I20240501 14:10:28.866185 19505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98af8b06b9324e3fa7cdb898b7b9fab9" candidate_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a0be6ef2144b7fb247ba9d3f051d66"
I20240501 14:10:28.866295 19505 raft_consensus.cc:3037] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:28.866254 19638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98af8b06b9324e3fa7cdb898b7b9fab9" candidate_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec8cde17a00645e7b38d5d29f7425367"
I20240501 14:10:28.866423 19638 raft_consensus.cc:3037] T 98af8b06b9324e3fa7cdb898b7b9fab9 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:28.867378 19505 raft_consensus.cc:2454] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb1e6bc2d07b4ce3b622a66f14c93bdc in term 1.
I20240501 14:10:28.867450 19638 raft_consensus.cc:2454] T 98af8b06b9324e3fa7cdb898b7b9fab9 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb1e6bc2d07b4ce3b622a66f14c93bdc in term 1.
I20240501 14:10:28.867660 19310 leader_election.cc:302] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99a0be6ef2144b7fb247ba9d3f051d66, fb1e6bc2d07b4ce3b622a66f14c93bdc; no voters:
I20240501 14:10:28.867806 20499 raft_consensus.cc:2781] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:28.867899 20499 raft_consensus.cc:686] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Becoming Leader. State: Replica: fb1e6bc2d07b4ce3b622a66f14c93bdc, State: Running, Role: LEADER
I20240501 14:10:28.868062 20499 consensus_queue.cc:237] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:28.869668 19231 catalog_manager.cc:5484] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc reported cstate change: term changed from 0 to 1, leader changed from <none> to fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129). New cstate: current_term: 1 leader_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } }
I20240501 14:10:28.992574 20386 raft_consensus.cc:482] T bb291a0c743c4e638f79da36cc77faa3 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:28.992702 20386 raft_consensus.cc:504] T bb291a0c743c4e638f79da36cc77faa3 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:28.993052 20386 leader_election.cc:288] T bb291a0c743c4e638f79da36cc77faa3 P da3c4e948ff747678f18456c134d7ed4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201), fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847)
I20240501 14:10:28.993441 19638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb291a0c743c4e638f79da36cc77faa3" candidate_uuid: "da3c4e948ff747678f18456c134d7ed4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec8cde17a00645e7b38d5d29f7425367" is_pre_election: true
I20240501 14:10:28.993450 19374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb291a0c743c4e638f79da36cc77faa3" candidate_uuid: "da3c4e948ff747678f18456c134d7ed4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" is_pre_election: true
I20240501 14:10:28.993583 19638 raft_consensus.cc:2454] T bb291a0c743c4e638f79da36cc77faa3 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da3c4e948ff747678f18456c134d7ed4 in term 0.
I20240501 14:10:28.993594 19374 raft_consensus.cc:2454] T bb291a0c743c4e638f79da36cc77faa3 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da3c4e948ff747678f18456c134d7ed4 in term 0.
I20240501 14:10:28.993865 19703 leader_election.cc:302] T bb291a0c743c4e638f79da36cc77faa3 P da3c4e948ff747678f18456c134d7ed4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da3c4e948ff747678f18456c134d7ed4, ec8cde17a00645e7b38d5d29f7425367; no voters:
I20240501 14:10:28.994071 20386 raft_consensus.cc:2781] T bb291a0c743c4e638f79da36cc77faa3 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:28.994145 20386 raft_consensus.cc:482] T bb291a0c743c4e638f79da36cc77faa3 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:28.994197 20386 raft_consensus.cc:3037] T bb291a0c743c4e638f79da36cc77faa3 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:28.995131 20386 raft_consensus.cc:504] T bb291a0c743c4e638f79da36cc77faa3 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:28.995457 20386 leader_election.cc:288] T bb291a0c743c4e638f79da36cc77faa3 P da3c4e948ff747678f18456c134d7ed4 [CANDIDATE]: Term 1 election: Requested vote from peers ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201), fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847)
I20240501 14:10:28.995787 19638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb291a0c743c4e638f79da36cc77faa3" candidate_uuid: "da3c4e948ff747678f18456c134d7ed4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec8cde17a00645e7b38d5d29f7425367"
I20240501 14:10:28.995788 19374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb291a0c743c4e638f79da36cc77faa3" candidate_uuid: "da3c4e948ff747678f18456c134d7ed4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc"
I20240501 14:10:28.995888 19374 raft_consensus.cc:3037] T bb291a0c743c4e638f79da36cc77faa3 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:28.995895 19638 raft_consensus.cc:3037] T bb291a0c743c4e638f79da36cc77faa3 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:28.996675 19638 raft_consensus.cc:2454] T bb291a0c743c4e638f79da36cc77faa3 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da3c4e948ff747678f18456c134d7ed4 in term 1.
I20240501 14:10:28.996758 19374 raft_consensus.cc:2454] T bb291a0c743c4e638f79da36cc77faa3 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da3c4e948ff747678f18456c134d7ed4 in term 1.
I20240501 14:10:28.996961 19703 leader_election.cc:302] T bb291a0c743c4e638f79da36cc77faa3 P da3c4e948ff747678f18456c134d7ed4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da3c4e948ff747678f18456c134d7ed4, ec8cde17a00645e7b38d5d29f7425367; no voters:
I20240501 14:10:28.997119 20386 raft_consensus.cc:2781] T bb291a0c743c4e638f79da36cc77faa3 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:28.997215 20386 raft_consensus.cc:686] T bb291a0c743c4e638f79da36cc77faa3 P da3c4e948ff747678f18456c134d7ed4 [term 1 LEADER]: Becoming Leader. State: Replica: da3c4e948ff747678f18456c134d7ed4, State: Running, Role: LEADER
I20240501 14:10:28.997380 20386 consensus_queue.cc:237] T bb291a0c743c4e638f79da36cc77faa3 P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:28.999015 19231 catalog_manager.cc:5484] T bb291a0c743c4e638f79da36cc77faa3 P da3c4e948ff747678f18456c134d7ed4 reported cstate change: term changed from 0 to 1, leader changed from <none> to da3c4e948ff747678f18456c134d7ed4 (127.7.39.132). New cstate: current_term: 1 leader_uuid: "da3c4e948ff747678f18456c134d7ed4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } }
I20240501 14:10:29.055732 20654 raft_consensus.cc:482] T 12ffa74f7a5b472aac57253693273cad P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:29.055879 20654 raft_consensus.cc:504] T 12ffa74f7a5b472aac57253693273cad P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:29.056325 20654 leader_election.cc:288] T 12ffa74f7a5b472aac57253693273cad P ec8cde17a00645e7b38d5d29f7425367 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847), da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357)
I20240501 14:10:29.056836 19769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12ffa74f7a5b472aac57253693273cad" candidate_uuid: "ec8cde17a00645e7b38d5d29f7425367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3c4e948ff747678f18456c134d7ed4" is_pre_election: true
I20240501 14:10:29.056836 19374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12ffa74f7a5b472aac57253693273cad" candidate_uuid: "ec8cde17a00645e7b38d5d29f7425367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" is_pre_election: true
I20240501 14:10:29.057009 19374 raft_consensus.cc:2454] T 12ffa74f7a5b472aac57253693273cad P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec8cde17a00645e7b38d5d29f7425367 in term 0.
I20240501 14:10:29.057009 19769 raft_consensus.cc:2454] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec8cde17a00645e7b38d5d29f7425367 in term 0.
I20240501 14:10:29.057341 19573 leader_election.cc:302] T 12ffa74f7a5b472aac57253693273cad P ec8cde17a00645e7b38d5d29f7425367 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da3c4e948ff747678f18456c134d7ed4, ec8cde17a00645e7b38d5d29f7425367; no voters:
I20240501 14:10:29.057549 20654 raft_consensus.cc:2781] T 12ffa74f7a5b472aac57253693273cad P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:29.057615 20654 raft_consensus.cc:482] T 12ffa74f7a5b472aac57253693273cad P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:29.057643 20654 raft_consensus.cc:3037] T 12ffa74f7a5b472aac57253693273cad P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:29.058501 20654 raft_consensus.cc:504] T 12ffa74f7a5b472aac57253693273cad P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:29.058835 20654 leader_election.cc:288] T 12ffa74f7a5b472aac57253693273cad P ec8cde17a00645e7b38d5d29f7425367 [CANDIDATE]: Term 1 election: Requested vote from peers fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847), da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357)
I20240501 14:10:29.059201 19769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12ffa74f7a5b472aac57253693273cad" candidate_uuid: "ec8cde17a00645e7b38d5d29f7425367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3c4e948ff747678f18456c134d7ed4"
I20240501 14:10:29.059202 19374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12ffa74f7a5b472aac57253693273cad" candidate_uuid: "ec8cde17a00645e7b38d5d29f7425367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc"
I20240501 14:10:29.059324 19769 raft_consensus.cc:3037] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:29.059334 19374 raft_consensus.cc:3037] T 12ffa74f7a5b472aac57253693273cad P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:29.060278 19374 raft_consensus.cc:2454] T 12ffa74f7a5b472aac57253693273cad P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec8cde17a00645e7b38d5d29f7425367 in term 1.
I20240501 14:10:29.060278 19769 raft_consensus.cc:2454] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec8cde17a00645e7b38d5d29f7425367 in term 1.
I20240501 14:10:29.060611 19573 leader_election.cc:302] T 12ffa74f7a5b472aac57253693273cad P ec8cde17a00645e7b38d5d29f7425367 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da3c4e948ff747678f18456c134d7ed4, ec8cde17a00645e7b38d5d29f7425367; no voters:
I20240501 14:10:29.060827 20654 raft_consensus.cc:2781] T 12ffa74f7a5b472aac57253693273cad P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:29.060951 20654 raft_consensus.cc:686] T 12ffa74f7a5b472aac57253693273cad P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Becoming Leader. State: Replica: ec8cde17a00645e7b38d5d29f7425367, State: Running, Role: LEADER
I20240501 14:10:29.061095 20654 consensus_queue.cc:237] T 12ffa74f7a5b472aac57253693273cad P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:29.062772 19231 catalog_manager.cc:5484] T 12ffa74f7a5b472aac57253693273cad P ec8cde17a00645e7b38d5d29f7425367 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131). New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } }
W20240501 14:10:29.182976 20720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:10:29.183296 20720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:10:29.186378 20720 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240501 14:10:29.276564 19374 raft_consensus.cc:1261] T bb291a0c743c4e638f79da36cc77faa3 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:29.277117 20386 consensus_queue.cc:1035] T bb291a0c743c4e638f79da36cc77faa3 P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:29.277267 19638 raft_consensus.cc:1261] T bb291a0c743c4e638f79da36cc77faa3 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:29.277638 20386 consensus_queue.cc:1035] T bb291a0c743c4e638f79da36cc77faa3 P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:29.278296 19638 raft_consensus.cc:1261] T 98af8b06b9324e3fa7cdb898b7b9fab9 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:29.278767 20511 consensus_queue.cc:1035] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:29.279732 19375 raft_consensus.cc:1261] T 12ffa74f7a5b472aac57253693273cad P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:29.280200 20652 consensus_queue.cc:1035] T 12ffa74f7a5b472aac57253693273cad P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:29.282279 19505 raft_consensus.cc:1261] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:29.284912 20499 consensus_queue.cc:1035] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:29.287241 19769 raft_consensus.cc:1261] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:29.288534 20652 consensus_queue.cc:1035] T 12ffa74f7a5b472aac57253693273cad P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:29.289156 20705 mvcc.cc:205] Tried to move back new op lower bound from 7022889489512689664 to 7022889488374734848. Current Snapshot: MvccSnapshot[applied={T|T < 7022889489512689664}]
I20240501 14:10:29.291607 20578 mvcc.cc:205] Tried to move back new op lower bound from 7022889489512689664 to 7022889488374734848. Current Snapshot: MvccSnapshot[applied={T|T < 7022889489512689664}]
I20240501 14:10:29.306737 20568 mvcc.cc:205] Tried to move back new op lower bound from 7022889489512689664 to 7022889488374734848. Current Snapshot: MvccSnapshot[applied={T|T < 7022889489512689664}]
INSERT report
rows total: 2000
time total: 114.846 ms
time per row: 0.057423 ms
I20240501 14:10:29.353634 19226 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45968:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:10:29.359723 19485 tablet_service.cc:1467] Processing CreateTablet for tablet d63afd081986473aa2715d074b123a93 (DEFAULT_TABLE table=rebalance_test_table_6 [id=afc36d383e3e4795ad11804e9bebcb51]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:29.359851 19486 tablet_service.cc:1467] Processing CreateTablet for tablet a9edf1764b55420ea5d519e07ae1d5e0 (DEFAULT_TABLE table=rebalance_test_table_6 [id=afc36d383e3e4795ad11804e9bebcb51]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:29.360066 19486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9edf1764b55420ea5d519e07ae1d5e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:29.360306 19749 tablet_service.cc:1467] Processing CreateTablet for tablet d63afd081986473aa2715d074b123a93 (DEFAULT_TABLE table=rebalance_test_table_6 [id=afc36d383e3e4795ad11804e9bebcb51]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:29.360611 19749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d63afd081986473aa2715d074b123a93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:29.362450 19485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d63afd081986473aa2715d074b123a93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:29.363147 20767 tablet_bootstrap.cc:492] T d63afd081986473aa2715d074b123a93 P da3c4e948ff747678f18456c134d7ed4: Bootstrap starting.
I20240501 14:10:29.363197 19356 tablet_service.cc:1467] Processing CreateTablet for tablet d63afd081986473aa2715d074b123a93 (DEFAULT_TABLE table=rebalance_test_table_6 [id=afc36d383e3e4795ad11804e9bebcb51]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:29.363294 19355 tablet_service.cc:1467] Processing CreateTablet for tablet ad7c255baee74e629e2976a942a4174f (DEFAULT_TABLE table=rebalance_test_table_6 [id=afc36d383e3e4795ad11804e9bebcb51]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:29.363479 19356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d63afd081986473aa2715d074b123a93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:29.363888 19355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad7c255baee74e629e2976a942a4174f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:29.364171 20767 tablet_bootstrap.cc:654] T d63afd081986473aa2715d074b123a93 P da3c4e948ff747678f18456c134d7ed4: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:29.364630 19617 tablet_service.cc:1467] Processing CreateTablet for tablet a9edf1764b55420ea5d519e07ae1d5e0 (DEFAULT_TABLE table=rebalance_test_table_6 [id=afc36d383e3e4795ad11804e9bebcb51]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:29.364912 19617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9edf1764b55420ea5d519e07ae1d5e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:29.365646 20769 tablet_bootstrap.cc:492] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Bootstrap starting.
I20240501 14:10:29.366411 20767 tablet_bootstrap.cc:492] T d63afd081986473aa2715d074b123a93 P da3c4e948ff747678f18456c134d7ed4: No bootstrap required, opened a new log
I20240501 14:10:29.366503 20767 ts_tablet_manager.cc:1367] T d63afd081986473aa2715d074b123a93 P da3c4e948ff747678f18456c134d7ed4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:10:29.366554 19749 tablet_service.cc:1467] Processing CreateTablet for tablet a9edf1764b55420ea5d519e07ae1d5e0 (DEFAULT_TABLE table=rebalance_test_table_6 [id=afc36d383e3e4795ad11804e9bebcb51]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:29.366693 19485 tablet_service.cc:1467] Processing CreateTablet for tablet ad7c255baee74e629e2976a942a4174f (DEFAULT_TABLE table=rebalance_test_table_6 [id=afc36d383e3e4795ad11804e9bebcb51]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:29.366773 19749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9edf1764b55420ea5d519e07ae1d5e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:29.366909 19485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad7c255baee74e629e2976a942a4174f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:29.367012 20767 raft_consensus.cc:348] T d63afd081986473aa2715d074b123a93 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:29.367187 20767 raft_consensus.cc:374] T d63afd081986473aa2715d074b123a93 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:29.367225 20767 raft_consensus.cc:729] T d63afd081986473aa2715d074b123a93 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3c4e948ff747678f18456c134d7ed4, State: Initialized, Role: FOLLOWER
I20240501 14:10:29.367391 20767 consensus_queue.cc:260] T d63afd081986473aa2715d074b123a93 P da3c4e948ff747678f18456c134d7ed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:29.368415 20766 tablet_bootstrap.cc:492] T a9edf1764b55420ea5d519e07ae1d5e0 P 99a0be6ef2144b7fb247ba9d3f051d66: Bootstrap starting.
I20240501 14:10:29.368947 19748 tablet_service.cc:1467] Processing CreateTablet for tablet ad7c255baee74e629e2976a942a4174f (DEFAULT_TABLE table=rebalance_test_table_6 [id=afc36d383e3e4795ad11804e9bebcb51]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:29.369171 19748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad7c255baee74e629e2976a942a4174f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:29.369302 20766 tablet_bootstrap.cc:654] T a9edf1764b55420ea5d519e07ae1d5e0 P 99a0be6ef2144b7fb247ba9d3f051d66: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:29.370446 20766 tablet_bootstrap.cc:492] T a9edf1764b55420ea5d519e07ae1d5e0 P 99a0be6ef2144b7fb247ba9d3f051d66: No bootstrap required, opened a new log
I20240501 14:10:29.370460 20770 tablet_bootstrap.cc:492] T a9edf1764b55420ea5d519e07ae1d5e0 P ec8cde17a00645e7b38d5d29f7425367: Bootstrap starting.
I20240501 14:10:29.370584 20766 ts_tablet_manager.cc:1367] T a9edf1764b55420ea5d519e07ae1d5e0 P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:10:29.370785 20767 ts_tablet_manager.cc:1398] T d63afd081986473aa2715d074b123a93 P da3c4e948ff747678f18456c134d7ed4: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:10:29.371026 20766 raft_consensus.cc:348] T a9edf1764b55420ea5d519e07ae1d5e0 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:29.371160 20766 raft_consensus.cc:374] T a9edf1764b55420ea5d519e07ae1d5e0 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:29.371193 20766 raft_consensus.cc:729] T a9edf1764b55420ea5d519e07ae1d5e0 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a0be6ef2144b7fb247ba9d3f051d66, State: Initialized, Role: FOLLOWER
I20240501 14:10:29.371351 20767 tablet_bootstrap.cc:492] T a9edf1764b55420ea5d519e07ae1d5e0 P da3c4e948ff747678f18456c134d7ed4: Bootstrap starting.
I20240501 14:10:29.371330 20766 consensus_queue.cc:260] T a9edf1764b55420ea5d519e07ae1d5e0 P 99a0be6ef2144b7fb247ba9d3f051d66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:29.372098 20770 tablet_bootstrap.cc:654] T a9edf1764b55420ea5d519e07ae1d5e0 P ec8cde17a00645e7b38d5d29f7425367: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:29.372340 20766 ts_tablet_manager.cc:1398] T a9edf1764b55420ea5d519e07ae1d5e0 P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:10:29.372380 20767 tablet_bootstrap.cc:654] T a9edf1764b55420ea5d519e07ae1d5e0 P da3c4e948ff747678f18456c134d7ed4: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:29.372504 20766 tablet_bootstrap.cc:492] T d63afd081986473aa2715d074b123a93 P 99a0be6ef2144b7fb247ba9d3f051d66: Bootstrap starting.
I20240501 14:10:29.373365 20766 tablet_bootstrap.cc:654] T d63afd081986473aa2715d074b123a93 P 99a0be6ef2144b7fb247ba9d3f051d66: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:29.373435 20770 tablet_bootstrap.cc:492] T a9edf1764b55420ea5d519e07ae1d5e0 P ec8cde17a00645e7b38d5d29f7425367: No bootstrap required, opened a new log
I20240501 14:10:29.373543 20770 ts_tablet_manager.cc:1367] T a9edf1764b55420ea5d519e07ae1d5e0 P ec8cde17a00645e7b38d5d29f7425367: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:10:29.373872 20769 tablet_bootstrap.cc:654] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:29.374117 20770 raft_consensus.cc:348] T a9edf1764b55420ea5d519e07ae1d5e0 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:29.374255 20770 raft_consensus.cc:374] T a9edf1764b55420ea5d519e07ae1d5e0 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:29.374305 20770 raft_consensus.cc:729] T a9edf1764b55420ea5d519e07ae1d5e0 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec8cde17a00645e7b38d5d29f7425367, State: Initialized, Role: FOLLOWER
I20240501 14:10:29.374460 20770 consensus_queue.cc:260] T a9edf1764b55420ea5d519e07ae1d5e0 P ec8cde17a00645e7b38d5d29f7425367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:29.374807 20770 ts_tablet_manager.cc:1398] T a9edf1764b55420ea5d519e07ae1d5e0 P ec8cde17a00645e7b38d5d29f7425367: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:10:29.375027 20767 tablet_bootstrap.cc:492] T a9edf1764b55420ea5d519e07ae1d5e0 P da3c4e948ff747678f18456c134d7ed4: No bootstrap required, opened a new log
I20240501 14:10:29.375150 20767 ts_tablet_manager.cc:1367] T a9edf1764b55420ea5d519e07ae1d5e0 P da3c4e948ff747678f18456c134d7ed4: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:10:29.375847 20767 raft_consensus.cc:348] T a9edf1764b55420ea5d519e07ae1d5e0 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:29.375946 20769 tablet_bootstrap.cc:492] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc: No bootstrap required, opened a new log
I20240501 14:10:29.375983 20767 raft_consensus.cc:374] T a9edf1764b55420ea5d519e07ae1d5e0 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:29.376024 20767 raft_consensus.cc:729] T a9edf1764b55420ea5d519e07ae1d5e0 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3c4e948ff747678f18456c134d7ed4, State: Initialized, Role: FOLLOWER
I20240501 14:10:29.376060 20769 ts_tablet_manager.cc:1367] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 14:10:29.376129 20767 consensus_queue.cc:260] T a9edf1764b55420ea5d519e07ae1d5e0 P da3c4e948ff747678f18456c134d7ed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:29.376418 20767 ts_tablet_manager.cc:1398] T a9edf1764b55420ea5d519e07ae1d5e0 P da3c4e948ff747678f18456c134d7ed4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:10:29.376612 20769 raft_consensus.cc:348] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:29.376718 20769 raft_consensus.cc:374] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:29.376742 20769 raft_consensus.cc:729] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1e6bc2d07b4ce3b622a66f14c93bdc, State: Initialized, Role: FOLLOWER
I20240501 14:10:29.376857 20769 consensus_queue.cc:260] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:29.376924 20767 tablet_bootstrap.cc:492] T ad7c255baee74e629e2976a942a4174f P da3c4e948ff747678f18456c134d7ed4: Bootstrap starting.
I20240501 14:10:29.377130 20769 ts_tablet_manager.cc:1398] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:29.377352 20769 tablet_bootstrap.cc:492] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc: Bootstrap starting.
I20240501 14:10:29.377887 20767 tablet_bootstrap.cc:654] T ad7c255baee74e629e2976a942a4174f P da3c4e948ff747678f18456c134d7ed4: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:29.378248 20769 tablet_bootstrap.cc:654] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:29.378901 20767 tablet_bootstrap.cc:492] T ad7c255baee74e629e2976a942a4174f P da3c4e948ff747678f18456c134d7ed4: No bootstrap required, opened a new log
I20240501 14:10:29.379060 20767 ts_tablet_manager.cc:1367] T ad7c255baee74e629e2976a942a4174f P da3c4e948ff747678f18456c134d7ed4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:10:29.379650 20767 raft_consensus.cc:348] T ad7c255baee74e629e2976a942a4174f P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:29.379788 20767 raft_consensus.cc:374] T ad7c255baee74e629e2976a942a4174f P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:29.379864 20767 raft_consensus.cc:729] T ad7c255baee74e629e2976a942a4174f P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3c4e948ff747678f18456c134d7ed4, State: Initialized, Role: FOLLOWER
I20240501 14:10:29.380043 20767 consensus_queue.cc:260] T ad7c255baee74e629e2976a942a4174f P da3c4e948ff747678f18456c134d7ed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:29.380182 20766 tablet_bootstrap.cc:492] T d63afd081986473aa2715d074b123a93 P 99a0be6ef2144b7fb247ba9d3f051d66: No bootstrap required, opened a new log
I20240501 14:10:29.380256 20766 ts_tablet_manager.cc:1367] T d63afd081986473aa2715d074b123a93 P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20240501 14:10:29.380373 20769 tablet_bootstrap.cc:492] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc: No bootstrap required, opened a new log
I20240501 14:10:29.380463 20769 ts_tablet_manager.cc:1367] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:10:29.380656 20767 ts_tablet_manager.cc:1398] T ad7c255baee74e629e2976a942a4174f P da3c4e948ff747678f18456c134d7ed4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:10:29.380733 20766 raft_consensus.cc:348] T d63afd081986473aa2715d074b123a93 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:29.380838 20766 raft_consensus.cc:374] T d63afd081986473aa2715d074b123a93 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:29.380879 20766 raft_consensus.cc:729] T d63afd081986473aa2715d074b123a93 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a0be6ef2144b7fb247ba9d3f051d66, State: Initialized, Role: FOLLOWER
I20240501 14:10:29.380934 20769 raft_consensus.cc:348] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:29.381053 20769 raft_consensus.cc:374] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:29.381013 20766 consensus_queue.cc:260] T d63afd081986473aa2715d074b123a93 P 99a0be6ef2144b7fb247ba9d3f051d66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:29.381098 20769 raft_consensus.cc:729] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1e6bc2d07b4ce3b622a66f14c93bdc, State: Initialized, Role: FOLLOWER
I20240501 14:10:29.381227 20769 consensus_queue.cc:260] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:29.381289 20766 ts_tablet_manager.cc:1398] T d63afd081986473aa2715d074b123a93 P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:10:29.381559 20769 ts_tablet_manager.cc:1398] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:29.382238 20766 tablet_bootstrap.cc:492] T ad7c255baee74e629e2976a942a4174f P 99a0be6ef2144b7fb247ba9d3f051d66: Bootstrap starting.
I20240501 14:10:29.383394 20766 tablet_bootstrap.cc:654] T ad7c255baee74e629e2976a942a4174f P 99a0be6ef2144b7fb247ba9d3f051d66: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:29.384730 20766 tablet_bootstrap.cc:492] T ad7c255baee74e629e2976a942a4174f P 99a0be6ef2144b7fb247ba9d3f051d66: No bootstrap required, opened a new log
I20240501 14:10:29.384820 20766 ts_tablet_manager.cc:1367] T ad7c255baee74e629e2976a942a4174f P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:10:29.385164 20766 raft_consensus.cc:348] T ad7c255baee74e629e2976a942a4174f P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:29.385247 20766 raft_consensus.cc:374] T ad7c255baee74e629e2976a942a4174f P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:29.385295 20766 raft_consensus.cc:729] T ad7c255baee74e629e2976a942a4174f P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a0be6ef2144b7fb247ba9d3f051d66, State: Initialized, Role: FOLLOWER
I20240501 14:10:29.385409 20766 consensus_queue.cc:260] T ad7c255baee74e629e2976a942a4174f P 99a0be6ef2144b7fb247ba9d3f051d66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:29.385733 20766 ts_tablet_manager.cc:1398] T ad7c255baee74e629e2976a942a4174f P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:10:29.493909 20750 raft_consensus.cc:482] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:29.494064 20750 raft_consensus.cc:504] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:29.494606 20750 leader_election.cc:288] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829), da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357)
I20240501 14:10:29.495137 19769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad7c255baee74e629e2976a942a4174f" candidate_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3c4e948ff747678f18456c134d7ed4" is_pre_election: true
I20240501 14:10:29.495143 19506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad7c255baee74e629e2976a942a4174f" candidate_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" is_pre_election: true
I20240501 14:10:29.495285 19769 raft_consensus.cc:2454] T ad7c255baee74e629e2976a942a4174f P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb1e6bc2d07b4ce3b622a66f14c93bdc in term 0.
I20240501 14:10:29.495316 19506 raft_consensus.cc:2454] T ad7c255baee74e629e2976a942a4174f P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb1e6bc2d07b4ce3b622a66f14c93bdc in term 0.
I20240501 14:10:29.495600 19311 leader_election.cc:302] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da3c4e948ff747678f18456c134d7ed4, fb1e6bc2d07b4ce3b622a66f14c93bdc; no voters:
I20240501 14:10:29.495811 20750 raft_consensus.cc:2781] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:29.495869 20750 raft_consensus.cc:482] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:29.495924 20750 raft_consensus.cc:3037] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:29.496793 20750 raft_consensus.cc:504] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:29.497119 20750 leader_election.cc:288] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc [CANDIDATE]: Term 1 election: Requested vote from peers 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829), da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357)
I20240501 14:10:29.497449 19769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad7c255baee74e629e2976a942a4174f" candidate_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3c4e948ff747678f18456c134d7ed4"
I20240501 14:10:29.497433 19506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad7c255baee74e629e2976a942a4174f" candidate_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a0be6ef2144b7fb247ba9d3f051d66"
I20240501 14:10:29.497558 19506 raft_consensus.cc:3037] T ad7c255baee74e629e2976a942a4174f P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:29.497557 19769 raft_consensus.cc:3037] T ad7c255baee74e629e2976a942a4174f P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:29.498275 19769 raft_consensus.cc:2454] T ad7c255baee74e629e2976a942a4174f P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb1e6bc2d07b4ce3b622a66f14c93bdc in term 1.
I20240501 14:10:29.498617 19506 raft_consensus.cc:2454] T ad7c255baee74e629e2976a942a4174f P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb1e6bc2d07b4ce3b622a66f14c93bdc in term 1.
I20240501 14:10:29.498701 19311 leader_election.cc:302] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da3c4e948ff747678f18456c134d7ed4, fb1e6bc2d07b4ce3b622a66f14c93bdc; no voters:
I20240501 14:10:29.498888 20750 raft_consensus.cc:2781] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:29.498965 20750 raft_consensus.cc:686] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Becoming Leader. State: Replica: fb1e6bc2d07b4ce3b622a66f14c93bdc, State: Running, Role: LEADER
I20240501 14:10:29.499140 20750 consensus_queue.cc:237] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:29.500741 19226 catalog_manager.cc:5484] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc reported cstate change: term changed from 0 to 1, leader changed from <none> to fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129). New cstate: current_term: 1 leader_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } }
I20240501 14:10:29.510957 20750 raft_consensus.cc:482] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:29.511093 20750 raft_consensus.cc:504] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:29.511469 20750 leader_election.cc:288] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829), da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357)
I20240501 14:10:29.511914 19769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d63afd081986473aa2715d074b123a93" candidate_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3c4e948ff747678f18456c134d7ed4" is_pre_election: true
I20240501 14:10:29.511961 19506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d63afd081986473aa2715d074b123a93" candidate_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" is_pre_election: true
I20240501 14:10:29.512102 19769 raft_consensus.cc:2454] T d63afd081986473aa2715d074b123a93 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb1e6bc2d07b4ce3b622a66f14c93bdc in term 0.
I20240501 14:10:29.512161 19506 raft_consensus.cc:2454] T d63afd081986473aa2715d074b123a93 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb1e6bc2d07b4ce3b622a66f14c93bdc in term 0.
I20240501 14:10:29.512451 19311 leader_election.cc:302] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da3c4e948ff747678f18456c134d7ed4, fb1e6bc2d07b4ce3b622a66f14c93bdc; no voters:
I20240501 14:10:29.512635 20750 raft_consensus.cc:2781] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:29.512691 20750 raft_consensus.cc:482] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:29.512719 20750 raft_consensus.cc:3037] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:29.513550 20750 raft_consensus.cc:504] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:29.513865 20750 leader_election.cc:288] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc [CANDIDATE]: Term 1 election: Requested vote from peers 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829), da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357)
I20240501 14:10:29.514189 19506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d63afd081986473aa2715d074b123a93" candidate_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a0be6ef2144b7fb247ba9d3f051d66"
I20240501 14:10:29.514253 19769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d63afd081986473aa2715d074b123a93" candidate_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3c4e948ff747678f18456c134d7ed4"
I20240501 14:10:29.514321 19506 raft_consensus.cc:3037] T d63afd081986473aa2715d074b123a93 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:29.514360 19769 raft_consensus.cc:3037] T d63afd081986473aa2715d074b123a93 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:29.515146 19769 raft_consensus.cc:2454] T d63afd081986473aa2715d074b123a93 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb1e6bc2d07b4ce3b622a66f14c93bdc in term 1.
I20240501 14:10:29.515409 19311 leader_election.cc:302] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da3c4e948ff747678f18456c134d7ed4, fb1e6bc2d07b4ce3b622a66f14c93bdc; no voters:
I20240501 14:10:29.515444 19506 raft_consensus.cc:2454] T d63afd081986473aa2715d074b123a93 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb1e6bc2d07b4ce3b622a66f14c93bdc in term 1.
I20240501 14:10:29.515573 20750 raft_consensus.cc:2781] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:29.515699 20750 raft_consensus.cc:686] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Becoming Leader. State: Replica: fb1e6bc2d07b4ce3b622a66f14c93bdc, State: Running, Role: LEADER
I20240501 14:10:29.515846 20750 consensus_queue.cc:237] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:29.517715 19226 catalog_manager.cc:5484] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc reported cstate change: term changed from 0 to 1, leader changed from <none> to fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129). New cstate: current_term: 1 leader_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } }
I20240501 14:10:29.723274 20372 raft_consensus.cc:482] T a9edf1764b55420ea5d519e07ae1d5e0 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:29.723412 20372 raft_consensus.cc:504] T a9edf1764b55420ea5d519e07ae1d5e0 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:29.723812 20372 leader_election.cc:288] T a9edf1764b55420ea5d519e07ae1d5e0 P 99a0be6ef2144b7fb247ba9d3f051d66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201), da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357)
I20240501 14:10:29.724135 19638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9edf1764b55420ea5d519e07ae1d5e0" candidate_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec8cde17a00645e7b38d5d29f7425367" is_pre_election: true
I20240501 14:10:29.724283 19638 raft_consensus.cc:2454] T a9edf1764b55420ea5d519e07ae1d5e0 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99a0be6ef2144b7fb247ba9d3f051d66 in term 0.
I20240501 14:10:29.724308 19769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9edf1764b55420ea5d519e07ae1d5e0" candidate_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3c4e948ff747678f18456c134d7ed4" is_pre_election: true
I20240501 14:10:29.724450 19769 raft_consensus.cc:2454] T a9edf1764b55420ea5d519e07ae1d5e0 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99a0be6ef2144b7fb247ba9d3f051d66 in term 0.
I20240501 14:10:29.724552 19441 leader_election.cc:302] T a9edf1764b55420ea5d519e07ae1d5e0 P 99a0be6ef2144b7fb247ba9d3f051d66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99a0be6ef2144b7fb247ba9d3f051d66, ec8cde17a00645e7b38d5d29f7425367; no voters:
I20240501 14:10:29.724690 20372 raft_consensus.cc:2781] T a9edf1764b55420ea5d519e07ae1d5e0 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:29.724722 20372 raft_consensus.cc:482] T a9edf1764b55420ea5d519e07ae1d5e0 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:29.724751 20372 raft_consensus.cc:3037] T a9edf1764b55420ea5d519e07ae1d5e0 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:29.728030 20372 raft_consensus.cc:504] T a9edf1764b55420ea5d519e07ae1d5e0 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:29.728439 20372 leader_election.cc:288] T a9edf1764b55420ea5d519e07ae1d5e0 P 99a0be6ef2144b7fb247ba9d3f051d66 [CANDIDATE]: Term 1 election: Requested vote from peers ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201), da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357)
I20240501 14:10:29.728755 19769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9edf1764b55420ea5d519e07ae1d5e0" candidate_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3c4e948ff747678f18456c134d7ed4"
I20240501 14:10:29.728754 19638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9edf1764b55420ea5d519e07ae1d5e0" candidate_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec8cde17a00645e7b38d5d29f7425367"
I20240501 14:10:29.728859 19638 raft_consensus.cc:3037] T a9edf1764b55420ea5d519e07ae1d5e0 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:29.728864 19769 raft_consensus.cc:3037] T a9edf1764b55420ea5d519e07ae1d5e0 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:29.729769 19638 raft_consensus.cc:2454] T a9edf1764b55420ea5d519e07ae1d5e0 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99a0be6ef2144b7fb247ba9d3f051d66 in term 1.
I20240501 14:10:29.729812 19769 raft_consensus.cc:2454] T a9edf1764b55420ea5d519e07ae1d5e0 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99a0be6ef2144b7fb247ba9d3f051d66 in term 1.
I20240501 14:10:29.730058 19441 leader_election.cc:302] T a9edf1764b55420ea5d519e07ae1d5e0 P 99a0be6ef2144b7fb247ba9d3f051d66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99a0be6ef2144b7fb247ba9d3f051d66, ec8cde17a00645e7b38d5d29f7425367; no voters:
I20240501 14:10:29.730257 20372 raft_consensus.cc:2781] T a9edf1764b55420ea5d519e07ae1d5e0 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:29.730468 20372 raft_consensus.cc:686] T a9edf1764b55420ea5d519e07ae1d5e0 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 LEADER]: Becoming Leader. State: Replica: 99a0be6ef2144b7fb247ba9d3f051d66, State: Running, Role: LEADER
I20240501 14:10:29.730614 20372 consensus_queue.cc:237] T a9edf1764b55420ea5d519e07ae1d5e0 P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:29.732537 19226 catalog_manager.cc:5484] T a9edf1764b55420ea5d519e07ae1d5e0 P 99a0be6ef2144b7fb247ba9d3f051d66 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130). New cstate: current_term: 1 leader_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } }
W20240501 14:10:29.846364 20783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:10:29.846623 20783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:10:29.849691 20783 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240501 14:10:29.902408 20750 consensus_queue.cc:1035] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:29.907169 20750 consensus_queue.cc:1035] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:29.930260 19506 raft_consensus.cc:1261] T d63afd081986473aa2715d074b123a93 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:29.931327 19638 raft_consensus.cc:1261] T a9edf1764b55420ea5d519e07ae1d5e0 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Refusing update from remote peer 99a0be6ef2144b7fb247ba9d3f051d66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:29.931911 20372 consensus_queue.cc:1035] T a9edf1764b55420ea5d519e07ae1d5e0 P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:29.937001 19769 raft_consensus.cc:1261] T d63afd081986473aa2715d074b123a93 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:29.937178 20511 consensus_queue.cc:1035] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:29.938035 19769 raft_consensus.cc:1261] T a9edf1764b55420ea5d519e07ae1d5e0 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Refusing update from remote peer 99a0be6ef2144b7fb247ba9d3f051d66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:29.938547 20781 consensus_queue.cc:1035] T a9edf1764b55420ea5d519e07ae1d5e0 P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:29.940670 20750 consensus_queue.cc:1035] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:29.944751 20810 mvcc.cc:205] Tried to move back new op lower bound from 7022889492195368960 to 7022889491378135040. Current Snapshot: MvccSnapshot[applied={T|T < 7022889492195368960}]
INSERT report
rows total: 2000
time total: 103.275 ms
time per row: 0.0516373 ms
I20240501 14:10:29.993826 19231 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45968:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:10:29.999161 19617 tablet_service.cc:1467] Processing CreateTablet for tablet cc8e888d46d34a22b6d4409ec2ab53a9 (DEFAULT_TABLE table=rebalance_test_table_7 [id=ed1a7ed87fa9480ba18a052130ecdc05]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:29.999289 19355 tablet_service.cc:1467] Processing CreateTablet for tablet cc8e888d46d34a22b6d4409ec2ab53a9 (DEFAULT_TABLE table=rebalance_test_table_7 [id=ed1a7ed87fa9480ba18a052130ecdc05]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:29.999375 19485 tablet_service.cc:1467] Processing CreateTablet for tablet cc8e888d46d34a22b6d4409ec2ab53a9 (DEFAULT_TABLE table=rebalance_test_table_7 [id=ed1a7ed87fa9480ba18a052130ecdc05]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:29.999488 19617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc8e888d46d34a22b6d4409ec2ab53a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:29.999590 19355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc8e888d46d34a22b6d4409ec2ab53a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:29.999619 19485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc8e888d46d34a22b6d4409ec2ab53a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:30.001201 20846 tablet_bootstrap.cc:492] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367: Bootstrap starting.
I20240501 14:10:30.001720 19356 tablet_service.cc:1467] Processing CreateTablet for tablet 0988c468757d49738765bdcb31794bd1 (DEFAULT_TABLE table=rebalance_test_table_7 [id=ed1a7ed87fa9480ba18a052130ecdc05]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:30.001971 19356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0988c468757d49738765bdcb31794bd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:30.002086 20846 tablet_bootstrap.cc:654] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:30.002405 19618 tablet_service.cc:1467] Processing CreateTablet for tablet 272eee3450cf4f48a4e1de2dcb28c0f7 (DEFAULT_TABLE table=rebalance_test_table_7 [id=ed1a7ed87fa9480ba18a052130ecdc05]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:30.002627 19618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 272eee3450cf4f48a4e1de2dcb28c0f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:30.003204 20846 tablet_bootstrap.cc:492] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367: No bootstrap required, opened a new log
I20240501 14:10:30.003279 20846 ts_tablet_manager.cc:1367] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:10:30.003727 20846 raft_consensus.cc:348] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:30.003842 20846 raft_consensus.cc:374] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:30.003877 20846 raft_consensus.cc:729] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec8cde17a00645e7b38d5d29f7425367, State: Initialized, Role: FOLLOWER
I20240501 14:10:30.004037 20846 consensus_queue.cc:260] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:30.004410 20846 ts_tablet_manager.cc:1398] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:30.004540 20846 tablet_bootstrap.cc:492] T 272eee3450cf4f48a4e1de2dcb28c0f7 P ec8cde17a00645e7b38d5d29f7425367: Bootstrap starting.
I20240501 14:10:30.004501 19485 tablet_service.cc:1467] Processing CreateTablet for tablet 272eee3450cf4f48a4e1de2dcb28c0f7 (DEFAULT_TABLE table=rebalance_test_table_7 [id=ed1a7ed87fa9480ba18a052130ecdc05]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:30.004681 19485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 272eee3450cf4f48a4e1de2dcb28c0f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:30.005049 19748 tablet_service.cc:1467] Processing CreateTablet for tablet 0988c468757d49738765bdcb31794bd1 (DEFAULT_TABLE table=rebalance_test_table_7 [id=ed1a7ed87fa9480ba18a052130ecdc05]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:30.005228 20846 tablet_bootstrap.cc:654] T 272eee3450cf4f48a4e1de2dcb28c0f7 P ec8cde17a00645e7b38d5d29f7425367: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:30.005332 19748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0988c468757d49738765bdcb31794bd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:30.006266 20846 tablet_bootstrap.cc:492] T 272eee3450cf4f48a4e1de2dcb28c0f7 P ec8cde17a00645e7b38d5d29f7425367: No bootstrap required, opened a new log
I20240501 14:10:30.006354 20846 ts_tablet_manager.cc:1367] T 272eee3450cf4f48a4e1de2dcb28c0f7 P ec8cde17a00645e7b38d5d29f7425367: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:10:30.006784 20846 raft_consensus.cc:348] T 272eee3450cf4f48a4e1de2dcb28c0f7 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:30.006881 20846 raft_consensus.cc:374] T 272eee3450cf4f48a4e1de2dcb28c0f7 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:30.006911 20846 raft_consensus.cc:729] T 272eee3450cf4f48a4e1de2dcb28c0f7 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec8cde17a00645e7b38d5d29f7425367, State: Initialized, Role: FOLLOWER
I20240501 14:10:30.007172 20846 consensus_queue.cc:260] T 272eee3450cf4f48a4e1de2dcb28c0f7 P ec8cde17a00645e7b38d5d29f7425367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:30.007472 20846 ts_tablet_manager.cc:1398] T 272eee3450cf4f48a4e1de2dcb28c0f7 P ec8cde17a00645e7b38d5d29f7425367: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:30.007716 20851 tablet_bootstrap.cc:492] T 0988c468757d49738765bdcb31794bd1 P da3c4e948ff747678f18456c134d7ed4: Bootstrap starting.
I20240501 14:10:30.007694 19749 tablet_service.cc:1467] Processing CreateTablet for tablet 272eee3450cf4f48a4e1de2dcb28c0f7 (DEFAULT_TABLE table=rebalance_test_table_7 [id=ed1a7ed87fa9480ba18a052130ecdc05]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:30.007907 19749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 272eee3450cf4f48a4e1de2dcb28c0f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:30.007980 19617 tablet_service.cc:1467] Processing CreateTablet for tablet 0988c468757d49738765bdcb31794bd1 (DEFAULT_TABLE table=rebalance_test_table_7 [id=ed1a7ed87fa9480ba18a052130ecdc05]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:30.008153 19617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0988c468757d49738765bdcb31794bd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:30.008563 20851 tablet_bootstrap.cc:654] T 0988c468757d49738765bdcb31794bd1 P da3c4e948ff747678f18456c134d7ed4: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:30.009711 20851 tablet_bootstrap.cc:492] T 0988c468757d49738765bdcb31794bd1 P da3c4e948ff747678f18456c134d7ed4: No bootstrap required, opened a new log
I20240501 14:10:30.009786 20851 ts_tablet_manager.cc:1367] T 0988c468757d49738765bdcb31794bd1 P da3c4e948ff747678f18456c134d7ed4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:10:30.009948 20846 tablet_bootstrap.cc:492] T 0988c468757d49738765bdcb31794bd1 P ec8cde17a00645e7b38d5d29f7425367: Bootstrap starting.
I20240501 14:10:30.010206 20851 raft_consensus.cc:348] T 0988c468757d49738765bdcb31794bd1 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:30.010314 20851 raft_consensus.cc:374] T 0988c468757d49738765bdcb31794bd1 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:30.010354 20851 raft_consensus.cc:729] T 0988c468757d49738765bdcb31794bd1 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3c4e948ff747678f18456c134d7ed4, State: Initialized, Role: FOLLOWER
I20240501 14:10:30.010557 20851 consensus_queue.cc:260] T 0988c468757d49738765bdcb31794bd1 P da3c4e948ff747678f18456c134d7ed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:30.010813 20846 tablet_bootstrap.cc:654] T 0988c468757d49738765bdcb31794bd1 P ec8cde17a00645e7b38d5d29f7425367: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:30.010869 20851 ts_tablet_manager.cc:1398] T 0988c468757d49738765bdcb31794bd1 P da3c4e948ff747678f18456c134d7ed4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:30.011016 20851 tablet_bootstrap.cc:492] T 272eee3450cf4f48a4e1de2dcb28c0f7 P da3c4e948ff747678f18456c134d7ed4: Bootstrap starting.
I20240501 14:10:30.011807 20848 tablet_bootstrap.cc:492] T cc8e888d46d34a22b6d4409ec2ab53a9 P 99a0be6ef2144b7fb247ba9d3f051d66: Bootstrap starting.
I20240501 14:10:30.011848 20851 tablet_bootstrap.cc:654] T 272eee3450cf4f48a4e1de2dcb28c0f7 P da3c4e948ff747678f18456c134d7ed4: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:30.012888 20848 tablet_bootstrap.cc:654] T cc8e888d46d34a22b6d4409ec2ab53a9 P 99a0be6ef2144b7fb247ba9d3f051d66: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:30.013100 20851 tablet_bootstrap.cc:492] T 272eee3450cf4f48a4e1de2dcb28c0f7 P da3c4e948ff747678f18456c134d7ed4: No bootstrap required, opened a new log
I20240501 14:10:30.013180 20851 ts_tablet_manager.cc:1367] T 272eee3450cf4f48a4e1de2dcb28c0f7 P da3c4e948ff747678f18456c134d7ed4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:10:30.013585 20847 tablet_bootstrap.cc:492] T cc8e888d46d34a22b6d4409ec2ab53a9 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Bootstrap starting.
I20240501 14:10:30.013613 20851 raft_consensus.cc:348] T 272eee3450cf4f48a4e1de2dcb28c0f7 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:30.013725 20851 raft_consensus.cc:374] T 272eee3450cf4f48a4e1de2dcb28c0f7 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:30.013756 20851 raft_consensus.cc:729] T 272eee3450cf4f48a4e1de2dcb28c0f7 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3c4e948ff747678f18456c134d7ed4, State: Initialized, Role: FOLLOWER
I20240501 14:10:30.013913 20851 consensus_queue.cc:260] T 272eee3450cf4f48a4e1de2dcb28c0f7 P da3c4e948ff747678f18456c134d7ed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:30.014284 20851 ts_tablet_manager.cc:1398] T 272eee3450cf4f48a4e1de2dcb28c0f7 P da3c4e948ff747678f18456c134d7ed4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:30.014456 20846 tablet_bootstrap.cc:492] T 0988c468757d49738765bdcb31794bd1 P ec8cde17a00645e7b38d5d29f7425367: No bootstrap required, opened a new log
I20240501 14:10:30.014532 20846 ts_tablet_manager.cc:1367] T 0988c468757d49738765bdcb31794bd1 P ec8cde17a00645e7b38d5d29f7425367: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:10:30.014780 20848 tablet_bootstrap.cc:492] T cc8e888d46d34a22b6d4409ec2ab53a9 P 99a0be6ef2144b7fb247ba9d3f051d66: No bootstrap required, opened a new log
I20240501 14:10:30.014878 20848 ts_tablet_manager.cc:1367] T cc8e888d46d34a22b6d4409ec2ab53a9 P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:10:30.014883 20846 raft_consensus.cc:348] T 0988c468757d49738765bdcb31794bd1 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:30.014997 20846 raft_consensus.cc:374] T 0988c468757d49738765bdcb31794bd1 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:30.015062 20846 raft_consensus.cc:729] T 0988c468757d49738765bdcb31794bd1 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec8cde17a00645e7b38d5d29f7425367, State: Initialized, Role: FOLLOWER
I20240501 14:10:30.015192 20846 consensus_queue.cc:260] T 0988c468757d49738765bdcb31794bd1 P ec8cde17a00645e7b38d5d29f7425367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:30.015410 20846 ts_tablet_manager.cc:1398] T 0988c468757d49738765bdcb31794bd1 P ec8cde17a00645e7b38d5d29f7425367: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:30.015409 20848 raft_consensus.cc:348] T cc8e888d46d34a22b6d4409ec2ab53a9 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:30.015853 20848 raft_consensus.cc:374] T cc8e888d46d34a22b6d4409ec2ab53a9 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:30.015933 20848 raft_consensus.cc:729] T cc8e888d46d34a22b6d4409ec2ab53a9 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a0be6ef2144b7fb247ba9d3f051d66, State: Initialized, Role: FOLLOWER
I20240501 14:10:30.016063 20848 consensus_queue.cc:260] T cc8e888d46d34a22b6d4409ec2ab53a9 P 99a0be6ef2144b7fb247ba9d3f051d66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:30.016448 20848 ts_tablet_manager.cc:1398] T cc8e888d46d34a22b6d4409ec2ab53a9 P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:10:30.016630 20848 tablet_bootstrap.cc:492] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66: Bootstrap starting.
I20240501 14:10:30.017122 20847 tablet_bootstrap.cc:654] T cc8e888d46d34a22b6d4409ec2ab53a9 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:30.017720 20848 tablet_bootstrap.cc:654] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:30.018638 20847 tablet_bootstrap.cc:492] T cc8e888d46d34a22b6d4409ec2ab53a9 P fb1e6bc2d07b4ce3b622a66f14c93bdc: No bootstrap required, opened a new log
I20240501 14:10:30.018735 20847 ts_tablet_manager.cc:1367] T cc8e888d46d34a22b6d4409ec2ab53a9 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:10:30.018841 20848 tablet_bootstrap.cc:492] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66: No bootstrap required, opened a new log
I20240501 14:10:30.018929 20848 ts_tablet_manager.cc:1367] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:10:30.019225 20847 raft_consensus.cc:348] T cc8e888d46d34a22b6d4409ec2ab53a9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:30.019335 20847 raft_consensus.cc:374] T cc8e888d46d34a22b6d4409ec2ab53a9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:30.019356 20847 raft_consensus.cc:729] T cc8e888d46d34a22b6d4409ec2ab53a9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1e6bc2d07b4ce3b622a66f14c93bdc, State: Initialized, Role: FOLLOWER
I20240501 14:10:30.019318 20848 raft_consensus.cc:348] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:30.019415 20848 raft_consensus.cc:374] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:30.019481 20848 raft_consensus.cc:729] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a0be6ef2144b7fb247ba9d3f051d66, State: Initialized, Role: FOLLOWER
I20240501 14:10:30.019470 20847 consensus_queue.cc:260] T cc8e888d46d34a22b6d4409ec2ab53a9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:30.019603 20848 consensus_queue.cc:260] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:30.019742 20847 ts_tablet_manager.cc:1398] T cc8e888d46d34a22b6d4409ec2ab53a9 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:30.019881 20848 ts_tablet_manager.cc:1398] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:10:30.019912 20847 tablet_bootstrap.cc:492] T 0988c468757d49738765bdcb31794bd1 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Bootstrap starting.
I20240501 14:10:30.020707 20847 tablet_bootstrap.cc:654] T 0988c468757d49738765bdcb31794bd1 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:30.021816 20847 tablet_bootstrap.cc:492] T 0988c468757d49738765bdcb31794bd1 P fb1e6bc2d07b4ce3b622a66f14c93bdc: No bootstrap required, opened a new log
I20240501 14:10:30.021898 20847 ts_tablet_manager.cc:1367] T 0988c468757d49738765bdcb31794bd1 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:10:30.022328 20847 raft_consensus.cc:348] T 0988c468757d49738765bdcb31794bd1 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:30.022451 20847 raft_consensus.cc:374] T 0988c468757d49738765bdcb31794bd1 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:30.022492 20847 raft_consensus.cc:729] T 0988c468757d49738765bdcb31794bd1 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1e6bc2d07b4ce3b622a66f14c93bdc, State: Initialized, Role: FOLLOWER
I20240501 14:10:30.022615 20847 consensus_queue.cc:260] T 0988c468757d49738765bdcb31794bd1 P fb1e6bc2d07b4ce3b622a66f14c93bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:30.022883 20847 ts_tablet_manager.cc:1398] T 0988c468757d49738765bdcb31794bd1 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:10:30.076484 20654 raft_consensus.cc:482] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:30.076642 20654 raft_consensus.cc:504] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:30.077071 20654 leader_election.cc:288] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829), fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847)
I20240501 14:10:30.077469 19506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc8e888d46d34a22b6d4409ec2ab53a9" candidate_uuid: "ec8cde17a00645e7b38d5d29f7425367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" is_pre_election: true
I20240501 14:10:30.077495 19375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc8e888d46d34a22b6d4409ec2ab53a9" candidate_uuid: "ec8cde17a00645e7b38d5d29f7425367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" is_pre_election: true
I20240501 14:10:30.077610 19506 raft_consensus.cc:2454] T cc8e888d46d34a22b6d4409ec2ab53a9 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec8cde17a00645e7b38d5d29f7425367 in term 0.
I20240501 14:10:30.077620 19375 raft_consensus.cc:2454] T cc8e888d46d34a22b6d4409ec2ab53a9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec8cde17a00645e7b38d5d29f7425367 in term 0.
I20240501 14:10:30.077924 19572 leader_election.cc:302] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99a0be6ef2144b7fb247ba9d3f051d66, ec8cde17a00645e7b38d5d29f7425367; no voters:
I20240501 14:10:30.078099 20654 raft_consensus.cc:2781] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:30.078168 20654 raft_consensus.cc:482] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:30.078193 20654 raft_consensus.cc:3037] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:30.079144 20654 raft_consensus.cc:504] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:30.079453 20654 leader_election.cc:288] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 [CANDIDATE]: Term 1 election: Requested vote from peers 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829), fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847)
I20240501 14:10:30.079831 19375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc8e888d46d34a22b6d4409ec2ab53a9" candidate_uuid: "ec8cde17a00645e7b38d5d29f7425367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc"
I20240501 14:10:30.079815 19506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc8e888d46d34a22b6d4409ec2ab53a9" candidate_uuid: "ec8cde17a00645e7b38d5d29f7425367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a0be6ef2144b7fb247ba9d3f051d66"
I20240501 14:10:30.079936 19375 raft_consensus.cc:3037] T cc8e888d46d34a22b6d4409ec2ab53a9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:30.079959 19506 raft_consensus.cc:3037] T cc8e888d46d34a22b6d4409ec2ab53a9 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:30.080840 19375 raft_consensus.cc:2454] T cc8e888d46d34a22b6d4409ec2ab53a9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec8cde17a00645e7b38d5d29f7425367 in term 1.
I20240501 14:10:30.080849 19506 raft_consensus.cc:2454] T cc8e888d46d34a22b6d4409ec2ab53a9 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec8cde17a00645e7b38d5d29f7425367 in term 1.
I20240501 14:10:30.081111 19571 leader_election.cc:302] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ec8cde17a00645e7b38d5d29f7425367, fb1e6bc2d07b4ce3b622a66f14c93bdc; no voters:
I20240501 14:10:30.081252 20654 raft_consensus.cc:2781] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:30.081444 20654 raft_consensus.cc:686] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Becoming Leader. State: Replica: ec8cde17a00645e7b38d5d29f7425367, State: Running, Role: LEADER
I20240501 14:10:30.081609 20654 consensus_queue.cc:237] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:30.083196 19231 catalog_manager.cc:5484] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131). New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } }
I20240501 14:10:30.144946 20781 raft_consensus.cc:482] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:30.145078 20781 raft_consensus.cc:504] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:30.145438 20781 leader_election.cc:288] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201), da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357)
I20240501 14:10:30.145887 19637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272eee3450cf4f48a4e1de2dcb28c0f7" candidate_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec8cde17a00645e7b38d5d29f7425367" is_pre_election: true
I20240501 14:10:30.145884 19768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272eee3450cf4f48a4e1de2dcb28c0f7" candidate_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3c4e948ff747678f18456c134d7ed4" is_pre_election: true
I20240501 14:10:30.146023 19768 raft_consensus.cc:2454] T 272eee3450cf4f48a4e1de2dcb28c0f7 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99a0be6ef2144b7fb247ba9d3f051d66 in term 0.
I20240501 14:10:30.146034 19637 raft_consensus.cc:2454] T 272eee3450cf4f48a4e1de2dcb28c0f7 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99a0be6ef2144b7fb247ba9d3f051d66 in term 0.
I20240501 14:10:30.146312 19441 leader_election.cc:302] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99a0be6ef2144b7fb247ba9d3f051d66, ec8cde17a00645e7b38d5d29f7425367; no voters:
I20240501 14:10:30.146464 20781 raft_consensus.cc:2781] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:30.146523 20781 raft_consensus.cc:482] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:30.146576 20781 raft_consensus.cc:3037] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:30.147428 20781 raft_consensus.cc:504] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:30.147739 20781 leader_election.cc:288] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 [CANDIDATE]: Term 1 election: Requested vote from peers ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201), da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357)
I20240501 14:10:30.148010 19637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272eee3450cf4f48a4e1de2dcb28c0f7" candidate_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec8cde17a00645e7b38d5d29f7425367"
I20240501 14:10:30.148061 19768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "272eee3450cf4f48a4e1de2dcb28c0f7" candidate_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3c4e948ff747678f18456c134d7ed4"
I20240501 14:10:30.148113 19637 raft_consensus.cc:3037] T 272eee3450cf4f48a4e1de2dcb28c0f7 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:30.148177 19768 raft_consensus.cc:3037] T 272eee3450cf4f48a4e1de2dcb28c0f7 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:30.149001 19637 raft_consensus.cc:2454] T 272eee3450cf4f48a4e1de2dcb28c0f7 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99a0be6ef2144b7fb247ba9d3f051d66 in term 1.
I20240501 14:10:30.149024 19768 raft_consensus.cc:2454] T 272eee3450cf4f48a4e1de2dcb28c0f7 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99a0be6ef2144b7fb247ba9d3f051d66 in term 1.
I20240501 14:10:30.149299 19441 leader_election.cc:302] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99a0be6ef2144b7fb247ba9d3f051d66, ec8cde17a00645e7b38d5d29f7425367; no voters:
I20240501 14:10:30.149523 20781 raft_consensus.cc:2781] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:30.149596 20781 raft_consensus.cc:686] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 LEADER]: Becoming Leader. State: Replica: 99a0be6ef2144b7fb247ba9d3f051d66, State: Running, Role: LEADER
I20240501 14:10:30.149734 20781 consensus_queue.cc:237] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:30.151422 19231 catalog_manager.cc:5484] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130). New cstate: current_term: 1 leader_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } }
I20240501 14:10:30.251688 20835 raft_consensus.cc:482] T 0988c468757d49738765bdcb31794bd1 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:30.251828 20835 raft_consensus.cc:504] T 0988c468757d49738765bdcb31794bd1 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:30.252328 20835 leader_election.cc:288] T 0988c468757d49738765bdcb31794bd1 P fb1e6bc2d07b4ce3b622a66f14c93bdc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201), da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357)
I20240501 14:10:30.252746 19637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0988c468757d49738765bdcb31794bd1" candidate_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec8cde17a00645e7b38d5d29f7425367" is_pre_election: true
I20240501 14:10:30.252748 19768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0988c468757d49738765bdcb31794bd1" candidate_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3c4e948ff747678f18456c134d7ed4" is_pre_election: true
I20240501 14:10:30.252902 19768 raft_consensus.cc:2454] T 0988c468757d49738765bdcb31794bd1 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb1e6bc2d07b4ce3b622a66f14c93bdc in term 0.
I20240501 14:10:30.252902 19637 raft_consensus.cc:2454] T 0988c468757d49738765bdcb31794bd1 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb1e6bc2d07b4ce3b622a66f14c93bdc in term 0.
I20240501 14:10:30.253198 19311 leader_election.cc:302] T 0988c468757d49738765bdcb31794bd1 P fb1e6bc2d07b4ce3b622a66f14c93bdc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da3c4e948ff747678f18456c134d7ed4, fb1e6bc2d07b4ce3b622a66f14c93bdc; no voters:
I20240501 14:10:30.253394 20835 raft_consensus.cc:2781] T 0988c468757d49738765bdcb31794bd1 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:30.253484 20835 raft_consensus.cc:482] T 0988c468757d49738765bdcb31794bd1 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:30.253525 20835 raft_consensus.cc:3037] T 0988c468757d49738765bdcb31794bd1 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:30.254447 20835 raft_consensus.cc:504] T 0988c468757d49738765bdcb31794bd1 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:30.254739 20835 leader_election.cc:288] T 0988c468757d49738765bdcb31794bd1 P fb1e6bc2d07b4ce3b622a66f14c93bdc [CANDIDATE]: Term 1 election: Requested vote from peers ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201), da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357)
I20240501 14:10:30.255075 19637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0988c468757d49738765bdcb31794bd1" candidate_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec8cde17a00645e7b38d5d29f7425367"
I20240501 14:10:30.255122 19768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0988c468757d49738765bdcb31794bd1" candidate_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3c4e948ff747678f18456c134d7ed4"
I20240501 14:10:30.255192 19637 raft_consensus.cc:3037] T 0988c468757d49738765bdcb31794bd1 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:30.255244 19768 raft_consensus.cc:3037] T 0988c468757d49738765bdcb31794bd1 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:30.256165 19637 raft_consensus.cc:2454] T 0988c468757d49738765bdcb31794bd1 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb1e6bc2d07b4ce3b622a66f14c93bdc in term 1.
I20240501 14:10:30.256165 19768 raft_consensus.cc:2454] T 0988c468757d49738765bdcb31794bd1 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb1e6bc2d07b4ce3b622a66f14c93bdc in term 1.
I20240501 14:10:30.256443 19311 leader_election.cc:302] T 0988c468757d49738765bdcb31794bd1 P fb1e6bc2d07b4ce3b622a66f14c93bdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da3c4e948ff747678f18456c134d7ed4, fb1e6bc2d07b4ce3b622a66f14c93bdc; no voters:
I20240501 14:10:30.256623 20835 raft_consensus.cc:2781] T 0988c468757d49738765bdcb31794bd1 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:30.256779 20835 raft_consensus.cc:686] T 0988c468757d49738765bdcb31794bd1 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Becoming Leader. State: Replica: fb1e6bc2d07b4ce3b622a66f14c93bdc, State: Running, Role: LEADER
I20240501 14:10:30.257112 20835 consensus_queue.cc:237] T 0988c468757d49738765bdcb31794bd1 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:30.258945 19231 catalog_manager.cc:5484] T 0988c468757d49738765bdcb31794bd1 P fb1e6bc2d07b4ce3b622a66f14c93bdc reported cstate change: term changed from 0 to 1, leader changed from <none> to fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129). New cstate: current_term: 1 leader_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } }
W20240501 14:10:30.403349 20863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:10:30.403592 20863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:10:30.406536 20863 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240501 14:10:30.487797 19375 raft_consensus.cc:1261] T cc8e888d46d34a22b6d4409ec2ab53a9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:30.487906 19506 raft_consensus.cc:1261] T cc8e888d46d34a22b6d4409ec2ab53a9 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:30.488351 20859 consensus_queue.cc:1035] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:30.488554 20654 consensus_queue.cc:1035] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:30.491627 19769 raft_consensus.cc:1261] T 272eee3450cf4f48a4e1de2dcb28c0f7 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Refusing update from remote peer 99a0be6ef2144b7fb247ba9d3f051d66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:30.492067 19637 raft_consensus.cc:1261] T 272eee3450cf4f48a4e1de2dcb28c0f7 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Refusing update from remote peer 99a0be6ef2144b7fb247ba9d3f051d66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:30.492448 20372 consensus_queue.cc:1035] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:30.492734 20781 consensus_queue.cc:1035] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:30.508280 19769 raft_consensus.cc:1261] T 0988c468757d49738765bdcb31794bd1 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:30.508638 19637 raft_consensus.cc:1261] T 0988c468757d49738765bdcb31794bd1 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:30.509128 20834 consensus_queue.cc:1035] T 0988c468757d49738765bdcb31794bd1 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:30.509449 20835 consensus_queue.cc:1035] T 0988c468757d49738765bdcb31794bd1 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 117.131 ms
time per row: 0.0585655 ms
I20240501 14:10:30.564384 19229 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45968:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:10:30.569475 19485 tablet_service.cc:1467] Processing CreateTablet for tablet 2dcca6e0e6374520bd4522c80c110e97 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3866c35dd0e64d87b6049e5e14cb0d31]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:30.569821 19485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dcca6e0e6374520bd4522c80c110e97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:30.570044 19356 tablet_service.cc:1467] Processing CreateTablet for tablet 2dcca6e0e6374520bd4522c80c110e97 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3866c35dd0e64d87b6049e5e14cb0d31]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:30.570124 19618 tablet_service.cc:1467] Processing CreateTablet for tablet 11f64b59f83e458f8cf51fd5bcd36fee (DEFAULT_TABLE table=rebalance_test_table_8 [id=3866c35dd0e64d87b6049e5e14cb0d31]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:30.570324 19356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dcca6e0e6374520bd4522c80c110e97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:30.570358 19618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11f64b59f83e458f8cf51fd5bcd36fee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:30.571727 20918 tablet_bootstrap.cc:492] T 2dcca6e0e6374520bd4522c80c110e97 P 99a0be6ef2144b7fb247ba9d3f051d66: Bootstrap starting.
I20240501 14:10:30.571923 19617 tablet_service.cc:1467] Processing CreateTablet for tablet 2dcca6e0e6374520bd4522c80c110e97 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3866c35dd0e64d87b6049e5e14cb0d31]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:30.572136 19617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dcca6e0e6374520bd4522c80c110e97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:30.572625 20918 tablet_bootstrap.cc:654] T 2dcca6e0e6374520bd4522c80c110e97 P 99a0be6ef2144b7fb247ba9d3f051d66: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:30.572772 19749 tablet_service.cc:1467] Processing CreateTablet for tablet f6ca088ce3aa4b97aa54c8ce6d700921 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3866c35dd0e64d87b6049e5e14cb0d31]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:30.573042 19749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6ca088ce3aa4b97aa54c8ce6d700921. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:30.573984 19486 tablet_service.cc:1467] Processing CreateTablet for tablet f6ca088ce3aa4b97aa54c8ce6d700921 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3866c35dd0e64d87b6049e5e14cb0d31]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:30.574193 19486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6ca088ce3aa4b97aa54c8ce6d700921. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:30.574744 20922 tablet_bootstrap.cc:492] T f6ca088ce3aa4b97aa54c8ce6d700921 P da3c4e948ff747678f18456c134d7ed4: Bootstrap starting.
I20240501 14:10:30.575695 20920 tablet_bootstrap.cc:492] T 2dcca6e0e6374520bd4522c80c110e97 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Bootstrap starting.
I20240501 14:10:30.575737 19749 tablet_service.cc:1467] Processing CreateTablet for tablet 11f64b59f83e458f8cf51fd5bcd36fee (DEFAULT_TABLE table=rebalance_test_table_8 [id=3866c35dd0e64d87b6049e5e14cb0d31]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:30.575831 19356 tablet_service.cc:1467] Processing CreateTablet for tablet f6ca088ce3aa4b97aa54c8ce6d700921 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3866c35dd0e64d87b6049e5e14cb0d31]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:30.575981 19749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11f64b59f83e458f8cf51fd5bcd36fee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:30.576061 19355 tablet_service.cc:1467] Processing CreateTablet for tablet 11f64b59f83e458f8cf51fd5bcd36fee (DEFAULT_TABLE table=rebalance_test_table_8 [id=3866c35dd0e64d87b6049e5e14cb0d31]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:30.576443 20922 tablet_bootstrap.cc:654] T f6ca088ce3aa4b97aa54c8ce6d700921 P da3c4e948ff747678f18456c134d7ed4: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:30.576910 20920 tablet_bootstrap.cc:654] T 2dcca6e0e6374520bd4522c80c110e97 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:30.576078 19356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6ca088ce3aa4b97aa54c8ce6d700921. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:30.577476 20922 tablet_bootstrap.cc:492] T f6ca088ce3aa4b97aa54c8ce6d700921 P da3c4e948ff747678f18456c134d7ed4: No bootstrap required, opened a new log
I20240501 14:10:30.577572 20922 ts_tablet_manager.cc:1367] T f6ca088ce3aa4b97aa54c8ce6d700921 P da3c4e948ff747678f18456c134d7ed4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:10:30.577761 20919 tablet_bootstrap.cc:492] T 11f64b59f83e458f8cf51fd5bcd36fee P ec8cde17a00645e7b38d5d29f7425367: Bootstrap starting.
I20240501 14:10:30.578058 20922 raft_consensus.cc:348] T f6ca088ce3aa4b97aa54c8ce6d700921 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:30.578195 20922 raft_consensus.cc:374] T f6ca088ce3aa4b97aa54c8ce6d700921 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:30.578244 20922 raft_consensus.cc:729] T f6ca088ce3aa4b97aa54c8ce6d700921 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3c4e948ff747678f18456c134d7ed4, State: Initialized, Role: FOLLOWER
I20240501 14:10:30.578363 20922 consensus_queue.cc:260] T f6ca088ce3aa4b97aa54c8ce6d700921 P da3c4e948ff747678f18456c134d7ed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:30.578524 20918 tablet_bootstrap.cc:492] T 2dcca6e0e6374520bd4522c80c110e97 P 99a0be6ef2144b7fb247ba9d3f051d66: No bootstrap required, opened a new log
I20240501 14:10:30.578613 20918 ts_tablet_manager.cc:1367] T 2dcca6e0e6374520bd4522c80c110e97 P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:10:30.578642 20922 ts_tablet_manager.cc:1398] T f6ca088ce3aa4b97aa54c8ce6d700921 P da3c4e948ff747678f18456c134d7ed4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:30.578801 20922 tablet_bootstrap.cc:492] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4: Bootstrap starting.
I20240501 14:10:30.578886 19355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11f64b59f83e458f8cf51fd5bcd36fee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:30.579102 20918 raft_consensus.cc:348] T 2dcca6e0e6374520bd4522c80c110e97 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:30.579231 20918 raft_consensus.cc:374] T 2dcca6e0e6374520bd4522c80c110e97 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:30.579268 20918 raft_consensus.cc:729] T 2dcca6e0e6374520bd4522c80c110e97 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a0be6ef2144b7fb247ba9d3f051d66, State: Initialized, Role: FOLLOWER
I20240501 14:10:30.579416 20918 consensus_queue.cc:260] T 2dcca6e0e6374520bd4522c80c110e97 P 99a0be6ef2144b7fb247ba9d3f051d66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:30.579573 20919 tablet_bootstrap.cc:654] T 11f64b59f83e458f8cf51fd5bcd36fee P ec8cde17a00645e7b38d5d29f7425367: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:30.579645 20922 tablet_bootstrap.cc:654] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:30.579721 20918 ts_tablet_manager.cc:1398] T 2dcca6e0e6374520bd4522c80c110e97 P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:30.580413 20918 tablet_bootstrap.cc:492] T f6ca088ce3aa4b97aa54c8ce6d700921 P 99a0be6ef2144b7fb247ba9d3f051d66: Bootstrap starting.
I20240501 14:10:30.581179 20922 tablet_bootstrap.cc:492] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4: No bootstrap required, opened a new log
I20240501 14:10:30.581271 20918 tablet_bootstrap.cc:654] T f6ca088ce3aa4b97aa54c8ce6d700921 P 99a0be6ef2144b7fb247ba9d3f051d66: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:30.581272 20922 ts_tablet_manager.cc:1367] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:10:30.581686 20922 raft_consensus.cc:348] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:30.581804 20922 raft_consensus.cc:374] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:30.581840 20922 raft_consensus.cc:729] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3c4e948ff747678f18456c134d7ed4, State: Initialized, Role: FOLLOWER
I20240501 14:10:30.581949 20922 consensus_queue.cc:260] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:30.582165 20922 ts_tablet_manager.cc:1398] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:30.583288 20920 tablet_bootstrap.cc:492] T 2dcca6e0e6374520bd4522c80c110e97 P fb1e6bc2d07b4ce3b622a66f14c93bdc: No bootstrap required, opened a new log
I20240501 14:10:30.583388 20920 ts_tablet_manager.cc:1367] T 2dcca6e0e6374520bd4522c80c110e97 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:10:30.583477 20919 tablet_bootstrap.cc:492] T 11f64b59f83e458f8cf51fd5bcd36fee P ec8cde17a00645e7b38d5d29f7425367: No bootstrap required, opened a new log
I20240501 14:10:30.583583 20919 ts_tablet_manager.cc:1367] T 11f64b59f83e458f8cf51fd5bcd36fee P ec8cde17a00645e7b38d5d29f7425367: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:10:30.583851 20920 raft_consensus.cc:348] T 2dcca6e0e6374520bd4522c80c110e97 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:30.583981 20920 raft_consensus.cc:374] T 2dcca6e0e6374520bd4522c80c110e97 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:30.584017 20920 raft_consensus.cc:729] T 2dcca6e0e6374520bd4522c80c110e97 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1e6bc2d07b4ce3b622a66f14c93bdc, State: Initialized, Role: FOLLOWER
I20240501 14:10:30.584072 20919 raft_consensus.cc:348] T 11f64b59f83e458f8cf51fd5bcd36fee P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:30.584126 20920 consensus_queue.cc:260] T 2dcca6e0e6374520bd4522c80c110e97 P fb1e6bc2d07b4ce3b622a66f14c93bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:30.584182 20919 raft_consensus.cc:374] T 11f64b59f83e458f8cf51fd5bcd36fee P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:30.584218 20919 raft_consensus.cc:729] T 11f64b59f83e458f8cf51fd5bcd36fee P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec8cde17a00645e7b38d5d29f7425367, State: Initialized, Role: FOLLOWER
I20240501 14:10:30.584429 20920 ts_tablet_manager.cc:1398] T 2dcca6e0e6374520bd4522c80c110e97 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:30.584664 20920 tablet_bootstrap.cc:492] T 11f64b59f83e458f8cf51fd5bcd36fee P fb1e6bc2d07b4ce3b622a66f14c93bdc: Bootstrap starting.
I20240501 14:10:30.584328 20919 consensus_queue.cc:260] T 11f64b59f83e458f8cf51fd5bcd36fee P ec8cde17a00645e7b38d5d29f7425367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:30.585105 20919 ts_tablet_manager.cc:1398] T 11f64b59f83e458f8cf51fd5bcd36fee P ec8cde17a00645e7b38d5d29f7425367: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:30.585269 20919 tablet_bootstrap.cc:492] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367: Bootstrap starting.
I20240501 14:10:30.585748 20920 tablet_bootstrap.cc:654] T 11f64b59f83e458f8cf51fd5bcd36fee P fb1e6bc2d07b4ce3b622a66f14c93bdc: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:30.585922 20918 tablet_bootstrap.cc:492] T f6ca088ce3aa4b97aa54c8ce6d700921 P 99a0be6ef2144b7fb247ba9d3f051d66: No bootstrap required, opened a new log
I20240501 14:10:30.585994 20918 ts_tablet_manager.cc:1367] T f6ca088ce3aa4b97aa54c8ce6d700921 P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240501 14:10:30.586237 20919 tablet_bootstrap.cc:654] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:30.587608 20918 raft_consensus.cc:348] T f6ca088ce3aa4b97aa54c8ce6d700921 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:30.587711 20918 raft_consensus.cc:374] T f6ca088ce3aa4b97aa54c8ce6d700921 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:30.587747 20918 raft_consensus.cc:729] T f6ca088ce3aa4b97aa54c8ce6d700921 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a0be6ef2144b7fb247ba9d3f051d66, State: Initialized, Role: FOLLOWER
I20240501 14:10:30.587888 20918 consensus_queue.cc:260] T f6ca088ce3aa4b97aa54c8ce6d700921 P 99a0be6ef2144b7fb247ba9d3f051d66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:30.588181 20918 ts_tablet_manager.cc:1398] T f6ca088ce3aa4b97aa54c8ce6d700921 P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:10:30.588901 20920 tablet_bootstrap.cc:492] T 11f64b59f83e458f8cf51fd5bcd36fee P fb1e6bc2d07b4ce3b622a66f14c93bdc: No bootstrap required, opened a new log
I20240501 14:10:30.588954 20919 tablet_bootstrap.cc:492] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367: No bootstrap required, opened a new log
I20240501 14:10:30.588995 20920 ts_tablet_manager.cc:1367] T 11f64b59f83e458f8cf51fd5bcd36fee P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:10:30.589033 20919 ts_tablet_manager.cc:1367] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:10:30.589395 20919 raft_consensus.cc:348] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:30.589413 20920 raft_consensus.cc:348] T 11f64b59f83e458f8cf51fd5bcd36fee P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:30.589494 20919 raft_consensus.cc:374] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:30.589505 20920 raft_consensus.cc:374] T 11f64b59f83e458f8cf51fd5bcd36fee P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:30.589534 20919 raft_consensus.cc:729] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec8cde17a00645e7b38d5d29f7425367, State: Initialized, Role: FOLLOWER
I20240501 14:10:30.589540 20920 raft_consensus.cc:729] T 11f64b59f83e458f8cf51fd5bcd36fee P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1e6bc2d07b4ce3b622a66f14c93bdc, State: Initialized, Role: FOLLOWER
I20240501 14:10:30.589649 20920 consensus_queue.cc:260] T 11f64b59f83e458f8cf51fd5bcd36fee P fb1e6bc2d07b4ce3b622a66f14c93bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:30.589669 20919 consensus_queue.cc:260] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:30.589956 20920 ts_tablet_manager.cc:1398] T 11f64b59f83e458f8cf51fd5bcd36fee P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:30.590051 20919 ts_tablet_manager.cc:1398] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:30.590123 20920 tablet_bootstrap.cc:492] T f6ca088ce3aa4b97aa54c8ce6d700921 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Bootstrap starting.
I20240501 14:10:30.590976 20920 tablet_bootstrap.cc:654] T f6ca088ce3aa4b97aa54c8ce6d700921 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:30.592779 20920 tablet_bootstrap.cc:492] T f6ca088ce3aa4b97aa54c8ce6d700921 P fb1e6bc2d07b4ce3b622a66f14c93bdc: No bootstrap required, opened a new log
I20240501 14:10:30.592859 20920 ts_tablet_manager.cc:1367] T f6ca088ce3aa4b97aa54c8ce6d700921 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:10:30.593201 20920 raft_consensus.cc:348] T f6ca088ce3aa4b97aa54c8ce6d700921 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:30.593315 20920 raft_consensus.cc:374] T f6ca088ce3aa4b97aa54c8ce6d700921 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:30.593392 20920 raft_consensus.cc:729] T f6ca088ce3aa4b97aa54c8ce6d700921 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1e6bc2d07b4ce3b622a66f14c93bdc, State: Initialized, Role: FOLLOWER
I20240501 14:10:30.593528 20920 consensus_queue.cc:260] T f6ca088ce3aa4b97aa54c8ce6d700921 P fb1e6bc2d07b4ce3b622a66f14c93bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:30.593814 20920 ts_tablet_manager.cc:1398] T f6ca088ce3aa4b97aa54c8ce6d700921 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:30.661604 20654 raft_consensus.cc:482] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:30.661729 20654 raft_consensus.cc:504] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:30.662096 20654 leader_election.cc:288] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829), fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847)
I20240501 14:10:30.662436 19506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dcca6e0e6374520bd4522c80c110e97" candidate_uuid: "ec8cde17a00645e7b38d5d29f7425367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" is_pre_election: true
I20240501 14:10:30.662470 19375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dcca6e0e6374520bd4522c80c110e97" candidate_uuid: "ec8cde17a00645e7b38d5d29f7425367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" is_pre_election: true
I20240501 14:10:30.662577 19506 raft_consensus.cc:2454] T 2dcca6e0e6374520bd4522c80c110e97 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec8cde17a00645e7b38d5d29f7425367 in term 0.
I20240501 14:10:30.662597 19375 raft_consensus.cc:2454] T 2dcca6e0e6374520bd4522c80c110e97 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec8cde17a00645e7b38d5d29f7425367 in term 0.
I20240501 14:10:30.662849 19572 leader_election.cc:302] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99a0be6ef2144b7fb247ba9d3f051d66, ec8cde17a00645e7b38d5d29f7425367; no voters:
I20240501 14:10:30.662997 20654 raft_consensus.cc:2781] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:30.663079 20654 raft_consensus.cc:482] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:30.663123 20654 raft_consensus.cc:3037] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:30.663971 20654 raft_consensus.cc:504] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:30.664299 20654 leader_election.cc:288] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 [CANDIDATE]: Term 1 election: Requested vote from peers 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829), fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847)
I20240501 14:10:30.664558 19506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dcca6e0e6374520bd4522c80c110e97" candidate_uuid: "ec8cde17a00645e7b38d5d29f7425367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a0be6ef2144b7fb247ba9d3f051d66"
I20240501 14:10:30.664594 19375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dcca6e0e6374520bd4522c80c110e97" candidate_uuid: "ec8cde17a00645e7b38d5d29f7425367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc"
I20240501 14:10:30.664661 19506 raft_consensus.cc:3037] T 2dcca6e0e6374520bd4522c80c110e97 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:30.664695 19375 raft_consensus.cc:3037] T 2dcca6e0e6374520bd4522c80c110e97 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:30.665432 19506 raft_consensus.cc:2454] T 2dcca6e0e6374520bd4522c80c110e97 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec8cde17a00645e7b38d5d29f7425367 in term 1.
I20240501 14:10:30.665444 19375 raft_consensus.cc:2454] T 2dcca6e0e6374520bd4522c80c110e97 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec8cde17a00645e7b38d5d29f7425367 in term 1.
I20240501 14:10:30.665678 19572 leader_election.cc:302] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99a0be6ef2144b7fb247ba9d3f051d66, ec8cde17a00645e7b38d5d29f7425367; no voters:
I20240501 14:10:30.665900 20859 raft_consensus.cc:2781] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:30.666126 20859 raft_consensus.cc:686] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Becoming Leader. State: Replica: ec8cde17a00645e7b38d5d29f7425367, State: Running, Role: LEADER
I20240501 14:10:30.666263 20859 consensus_queue.cc:237] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:30.667977 19228 catalog_manager.cc:5484] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131). New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } }
I20240501 14:10:30.685317 20831 raft_consensus.cc:482] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:30.685438 20831 raft_consensus.cc:504] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:30.685799 20831 leader_election.cc:288] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201), fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847)
I20240501 14:10:30.686133 19637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11f64b59f83e458f8cf51fd5bcd36fee" candidate_uuid: "da3c4e948ff747678f18456c134d7ed4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec8cde17a00645e7b38d5d29f7425367" is_pre_election: true
I20240501 14:10:30.686157 19375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11f64b59f83e458f8cf51fd5bcd36fee" candidate_uuid: "da3c4e948ff747678f18456c134d7ed4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" is_pre_election: true
I20240501 14:10:30.686254 19637 raft_consensus.cc:2454] T 11f64b59f83e458f8cf51fd5bcd36fee P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da3c4e948ff747678f18456c134d7ed4 in term 0.
I20240501 14:10:30.686298 19375 raft_consensus.cc:2454] T 11f64b59f83e458f8cf51fd5bcd36fee P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da3c4e948ff747678f18456c134d7ed4 in term 0.
I20240501 14:10:30.686538 19703 leader_election.cc:302] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da3c4e948ff747678f18456c134d7ed4, ec8cde17a00645e7b38d5d29f7425367; no voters:
I20240501 14:10:30.686699 20831 raft_consensus.cc:2781] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:30.686753 20831 raft_consensus.cc:482] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:30.686816 20831 raft_consensus.cc:3037] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:30.687690 20831 raft_consensus.cc:504] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:30.687999 20831 leader_election.cc:288] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 [CANDIDATE]: Term 1 election: Requested vote from peers ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201), fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847)
I20240501 14:10:30.688349 19375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11f64b59f83e458f8cf51fd5bcd36fee" candidate_uuid: "da3c4e948ff747678f18456c134d7ed4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc"
I20240501 14:10:30.688499 19375 raft_consensus.cc:3037] T 11f64b59f83e458f8cf51fd5bcd36fee P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:30.688474 19637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11f64b59f83e458f8cf51fd5bcd36fee" candidate_uuid: "da3c4e948ff747678f18456c134d7ed4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec8cde17a00645e7b38d5d29f7425367"
I20240501 14:10:30.688601 19637 raft_consensus.cc:3037] T 11f64b59f83e458f8cf51fd5bcd36fee P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:30.689572 19375 raft_consensus.cc:2454] T 11f64b59f83e458f8cf51fd5bcd36fee P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da3c4e948ff747678f18456c134d7ed4 in term 1.
I20240501 14:10:30.689679 19637 raft_consensus.cc:2454] T 11f64b59f83e458f8cf51fd5bcd36fee P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da3c4e948ff747678f18456c134d7ed4 in term 1.
I20240501 14:10:30.689889 19702 leader_election.cc:302] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da3c4e948ff747678f18456c134d7ed4, fb1e6bc2d07b4ce3b622a66f14c93bdc; no voters:
I20240501 14:10:30.690097 20831 raft_consensus.cc:2781] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:30.690302 20831 raft_consensus.cc:686] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 [term 1 LEADER]: Becoming Leader. State: Replica: da3c4e948ff747678f18456c134d7ed4, State: Running, Role: LEADER
I20240501 14:10:30.690456 20831 consensus_queue.cc:237] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:10:30.692174 19228 catalog_manager.cc:5484] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 reported cstate change: term changed from 0 to 1, leader changed from <none> to da3c4e948ff747678f18456c134d7ed4 (127.7.39.132). New cstate: current_term: 1 leader_uuid: "da3c4e948ff747678f18456c134d7ed4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } }
I20240501 14:10:30.899010 20781 raft_consensus.cc:482] T f6ca088ce3aa4b97aa54c8ce6d700921 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:30.899169 20781 raft_consensus.cc:504] T f6ca088ce3aa4b97aa54c8ce6d700921 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:30.899530 20781 leader_election.cc:288] T f6ca088ce3aa4b97aa54c8ce6d700921 P 99a0be6ef2144b7fb247ba9d3f051d66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847), da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357)
I20240501 14:10:30.900002 19375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ca088ce3aa4b97aa54c8ce6d700921" candidate_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" is_pre_election: true
I20240501 14:10:30.900012 19768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ca088ce3aa4b97aa54c8ce6d700921" candidate_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3c4e948ff747678f18456c134d7ed4" is_pre_election: true
I20240501 14:10:30.900171 19768 raft_consensus.cc:2454] T f6ca088ce3aa4b97aa54c8ce6d700921 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99a0be6ef2144b7fb247ba9d3f051d66 in term 0.
I20240501 14:10:30.900171 19375 raft_consensus.cc:2454] T f6ca088ce3aa4b97aa54c8ce6d700921 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99a0be6ef2144b7fb247ba9d3f051d66 in term 0.
I20240501 14:10:30.900476 19442 leader_election.cc:302] T f6ca088ce3aa4b97aa54c8ce6d700921 P 99a0be6ef2144b7fb247ba9d3f051d66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 99a0be6ef2144b7fb247ba9d3f051d66, da3c4e948ff747678f18456c134d7ed4, fb1e6bc2d07b4ce3b622a66f14c93bdc; no voters:
I20240501 14:10:30.900703 20781 raft_consensus.cc:2781] T f6ca088ce3aa4b97aa54c8ce6d700921 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:30.900774 20781 raft_consensus.cc:482] T f6ca088ce3aa4b97aa54c8ce6d700921 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:30.900813 20781 raft_consensus.cc:3037] T f6ca088ce3aa4b97aa54c8ce6d700921 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:30.901649 20781 raft_consensus.cc:504] T f6ca088ce3aa4b97aa54c8ce6d700921 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:30.902016 20781 leader_election.cc:288] T f6ca088ce3aa4b97aa54c8ce6d700921 P 99a0be6ef2144b7fb247ba9d3f051d66 [CANDIDATE]: Term 1 election: Requested vote from peers fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847), da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357)
I20240501 14:10:30.902309 19375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ca088ce3aa4b97aa54c8ce6d700921" candidate_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc"
I20240501 14:10:30.902369 19768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ca088ce3aa4b97aa54c8ce6d700921" candidate_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3c4e948ff747678f18456c134d7ed4"
I20240501 14:10:30.902439 19375 raft_consensus.cc:3037] T f6ca088ce3aa4b97aa54c8ce6d700921 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:30.902468 19768 raft_consensus.cc:3037] T f6ca088ce3aa4b97aa54c8ce6d700921 P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:30.903410 19375 raft_consensus.cc:2454] T f6ca088ce3aa4b97aa54c8ce6d700921 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99a0be6ef2144b7fb247ba9d3f051d66 in term 1.
I20240501 14:10:30.903410 19768 raft_consensus.cc:2454] T f6ca088ce3aa4b97aa54c8ce6d700921 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99a0be6ef2144b7fb247ba9d3f051d66 in term 1.
I20240501 14:10:30.903694 19442 leader_election.cc:302] T f6ca088ce3aa4b97aa54c8ce6d700921 P 99a0be6ef2144b7fb247ba9d3f051d66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99a0be6ef2144b7fb247ba9d3f051d66, da3c4e948ff747678f18456c134d7ed4; no voters:
I20240501 14:10:30.903854 20781 raft_consensus.cc:2781] T f6ca088ce3aa4b97aa54c8ce6d700921 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:30.903928 20781 raft_consensus.cc:686] T f6ca088ce3aa4b97aa54c8ce6d700921 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 LEADER]: Becoming Leader. State: Replica: 99a0be6ef2144b7fb247ba9d3f051d66, State: Running, Role: LEADER
I20240501 14:10:30.904109 20781 consensus_queue.cc:237] T f6ca088ce3aa4b97aa54c8ce6d700921 P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:30.905742 19228 catalog_manager.cc:5484] T f6ca088ce3aa4b97aa54c8ce6d700921 P 99a0be6ef2144b7fb247ba9d3f051d66 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130). New cstate: current_term: 1 leader_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } }
W20240501 14:10:31.043349 20936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:10:31.043593 20936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:10:31.046752 20936 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240501 14:10:31.134487 19768 raft_consensus.cc:1261] T f6ca088ce3aa4b97aa54c8ce6d700921 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Refusing update from remote peer 99a0be6ef2144b7fb247ba9d3f051d66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:31.135726 19374 raft_consensus.cc:1261] T 2dcca6e0e6374520bd4522c80c110e97 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:31.136041 19506 raft_consensus.cc:1261] T 2dcca6e0e6374520bd4522c80c110e97 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:31.135746 19376 raft_consensus.cc:1261] T f6ca088ce3aa4b97aa54c8ce6d700921 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Refusing update from remote peer 99a0be6ef2144b7fb247ba9d3f051d66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:31.136600 20859 consensus_queue.cc:1035] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:31.137017 20831 consensus_queue.cc:1035] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:31.137822 20372 consensus_queue.cc:1035] T f6ca088ce3aa4b97aa54c8ce6d700921 P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:31.138769 20859 consensus_queue.cc:1035] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:31.141005 19637 raft_consensus.cc:1261] T 11f64b59f83e458f8cf51fd5bcd36fee P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:31.141315 20372 consensus_queue.cc:1035] T f6ca088ce3aa4b97aa54c8ce6d700921 P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:31.144716 20831 consensus_queue.cc:1035] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 119.104 ms
time per row: 0.0595522 ms
I20240501 14:10:31.206455 19232 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45968:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:10:31.212182 19617 tablet_service.cc:1467] Processing CreateTablet for tablet fe554c3145de46cf8070efcecffbaedb (DEFAULT_TABLE table=rebalance_test_table_9 [id=1fbbd460d13647c1a85b517926267158]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:31.212472 19749 tablet_service.cc:1467] Processing CreateTablet for tablet fe554c3145de46cf8070efcecffbaedb (DEFAULT_TABLE table=rebalance_test_table_9 [id=1fbbd460d13647c1a85b517926267158]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:31.212543 19617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe554c3145de46cf8070efcecffbaedb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:31.212754 19749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe554c3145de46cf8070efcecffbaedb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:31.212760 19486 tablet_service.cc:1467] Processing CreateTablet for tablet fe554c3145de46cf8070efcecffbaedb (DEFAULT_TABLE table=rebalance_test_table_9 [id=1fbbd460d13647c1a85b517926267158]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:31.213050 19486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe554c3145de46cf8070efcecffbaedb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:31.214723 19485 tablet_service.cc:1467] Processing CreateTablet for tablet 7eea58d4a33f422a9ada1e82d408577f (DEFAULT_TABLE table=rebalance_test_table_9 [id=1fbbd460d13647c1a85b517926267158]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:31.214794 21002 tablet_bootstrap.cc:492] T fe554c3145de46cf8070efcecffbaedb P da3c4e948ff747678f18456c134d7ed4: Bootstrap starting.
I20240501 14:10:31.214944 19485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7eea58d4a33f422a9ada1e82d408577f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:31.215183 21004 tablet_bootstrap.cc:492] T fe554c3145de46cf8070efcecffbaedb P 99a0be6ef2144b7fb247ba9d3f051d66: Bootstrap starting.
I20240501 14:10:31.215818 21002 tablet_bootstrap.cc:654] T fe554c3145de46cf8070efcecffbaedb P da3c4e948ff747678f18456c134d7ed4: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:31.215829 21004 tablet_bootstrap.cc:654] T fe554c3145de46cf8070efcecffbaedb P 99a0be6ef2144b7fb247ba9d3f051d66: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:31.216562 19748 tablet_service.cc:1467] Processing CreateTablet for tablet 7eea58d4a33f422a9ada1e82d408577f (DEFAULT_TABLE table=rebalance_test_table_9 [id=1fbbd460d13647c1a85b517926267158]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:31.216768 19748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7eea58d4a33f422a9ada1e82d408577f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:31.217308 21002 tablet_bootstrap.cc:492] T fe554c3145de46cf8070efcecffbaedb P da3c4e948ff747678f18456c134d7ed4: No bootstrap required, opened a new log
I20240501 14:10:31.217394 21002 ts_tablet_manager.cc:1367] T fe554c3145de46cf8070efcecffbaedb P da3c4e948ff747678f18456c134d7ed4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:10:31.217770 21002 raft_consensus.cc:348] T fe554c3145de46cf8070efcecffbaedb P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:31.217860 21002 raft_consensus.cc:374] T fe554c3145de46cf8070efcecffbaedb P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:31.217882 21002 raft_consensus.cc:729] T fe554c3145de46cf8070efcecffbaedb P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3c4e948ff747678f18456c134d7ed4, State: Initialized, Role: FOLLOWER
I20240501 14:10:31.217989 21002 consensus_queue.cc:260] T fe554c3145de46cf8070efcecffbaedb P da3c4e948ff747678f18456c134d7ed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:31.218302 21002 ts_tablet_manager.cc:1398] T fe554c3145de46cf8070efcecffbaedb P da3c4e948ff747678f18456c134d7ed4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:31.218770 21002 tablet_bootstrap.cc:492] T 7eea58d4a33f422a9ada1e82d408577f P da3c4e948ff747678f18456c134d7ed4: Bootstrap starting.
I20240501 14:10:31.218955 19356 tablet_service.cc:1467] Processing CreateTablet for tablet 10aa2feeb0324aa4a6221d2f2f3fc941 (DEFAULT_TABLE table=rebalance_test_table_9 [id=1fbbd460d13647c1a85b517926267158]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:31.219277 19356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10aa2feeb0324aa4a6221d2f2f3fc941. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:31.219446 19617 tablet_service.cc:1467] Processing CreateTablet for tablet 7eea58d4a33f422a9ada1e82d408577f (DEFAULT_TABLE table=rebalance_test_table_9 [id=1fbbd460d13647c1a85b517926267158]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:31.219691 19617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7eea58d4a33f422a9ada1e82d408577f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:31.219874 21002 tablet_bootstrap.cc:654] T 7eea58d4a33f422a9ada1e82d408577f P da3c4e948ff747678f18456c134d7ed4: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:31.222163 19485 tablet_service.cc:1467] Processing CreateTablet for tablet 10aa2feeb0324aa4a6221d2f2f3fc941 (DEFAULT_TABLE table=rebalance_test_table_9 [id=1fbbd460d13647c1a85b517926267158]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:31.222330 21003 tablet_bootstrap.cc:492] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367: Bootstrap starting.
I20240501 14:10:31.222394 19485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10aa2feeb0324aa4a6221d2f2f3fc941. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:31.222555 21004 tablet_bootstrap.cc:492] T fe554c3145de46cf8070efcecffbaedb P 99a0be6ef2144b7fb247ba9d3f051d66: No bootstrap required, opened a new log
I20240501 14:10:31.222635 21004 ts_tablet_manager.cc:1367] T fe554c3145de46cf8070efcecffbaedb P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240501 14:10:31.223033 21004 raft_consensus.cc:348] T fe554c3145de46cf8070efcecffbaedb P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:31.223165 21004 raft_consensus.cc:374] T fe554c3145de46cf8070efcecffbaedb P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:31.223196 21004 raft_consensus.cc:729] T fe554c3145de46cf8070efcecffbaedb P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a0be6ef2144b7fb247ba9d3f051d66, State: Initialized, Role: FOLLOWER
I20240501 14:10:31.223201 21008 tablet_bootstrap.cc:492] T 10aa2feeb0324aa4a6221d2f2f3fc941 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Bootstrap starting.
I20240501 14:10:31.223239 21003 tablet_bootstrap.cc:654] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:31.223348 21004 consensus_queue.cc:260] T fe554c3145de46cf8070efcecffbaedb P 99a0be6ef2144b7fb247ba9d3f051d66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:31.223716 21004 ts_tablet_manager.cc:1398] T fe554c3145de46cf8070efcecffbaedb P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:31.223948 21004 tablet_bootstrap.cc:492] T 7eea58d4a33f422a9ada1e82d408577f P 99a0be6ef2144b7fb247ba9d3f051d66: Bootstrap starting.
I20240501 14:10:31.224817 21004 tablet_bootstrap.cc:654] T 7eea58d4a33f422a9ada1e82d408577f P 99a0be6ef2144b7fb247ba9d3f051d66: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:31.224958 21008 tablet_bootstrap.cc:654] T 10aa2feeb0324aa4a6221d2f2f3fc941 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:31.226192 21008 tablet_bootstrap.cc:492] T 10aa2feeb0324aa4a6221d2f2f3fc941 P fb1e6bc2d07b4ce3b622a66f14c93bdc: No bootstrap required, opened a new log
I20240501 14:10:31.226220 21004 tablet_bootstrap.cc:492] T 7eea58d4a33f422a9ada1e82d408577f P 99a0be6ef2144b7fb247ba9d3f051d66: No bootstrap required, opened a new log
I20240501 14:10:31.226286 21004 ts_tablet_manager.cc:1367] T 7eea58d4a33f422a9ada1e82d408577f P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:10:31.226303 21008 ts_tablet_manager.cc:1367] T 10aa2feeb0324aa4a6221d2f2f3fc941 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:10:31.219447 19618 tablet_service.cc:1467] Processing CreateTablet for tablet 10aa2feeb0324aa4a6221d2f2f3fc941 (DEFAULT_TABLE table=rebalance_test_table_9 [id=1fbbd460d13647c1a85b517926267158]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:31.226812 19618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10aa2feeb0324aa4a6221d2f2f3fc941. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:31.226855 21004 raft_consensus.cc:348] T 7eea58d4a33f422a9ada1e82d408577f P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:31.226950 21004 raft_consensus.cc:374] T 7eea58d4a33f422a9ada1e82d408577f P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:31.226979 21004 raft_consensus.cc:729] T 7eea58d4a33f422a9ada1e82d408577f P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a0be6ef2144b7fb247ba9d3f051d66, State: Initialized, Role: FOLLOWER
I20240501 14:10:31.227149 21004 consensus_queue.cc:260] T 7eea58d4a33f422a9ada1e82d408577f P 99a0be6ef2144b7fb247ba9d3f051d66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:31.227141 21008 raft_consensus.cc:348] T 10aa2feeb0324aa4a6221d2f2f3fc941 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:31.227246 21008 raft_consensus.cc:374] T 10aa2feeb0324aa4a6221d2f2f3fc941 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:31.227281 21008 raft_consensus.cc:729] T 10aa2feeb0324aa4a6221d2f2f3fc941 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1e6bc2d07b4ce3b622a66f14c93bdc, State: Initialized, Role: FOLLOWER
I20240501 14:10:31.227399 21004 ts_tablet_manager.cc:1398] T 7eea58d4a33f422a9ada1e82d408577f P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:10:31.227403 21008 consensus_queue.cc:260] T 10aa2feeb0324aa4a6221d2f2f3fc941 P fb1e6bc2d07b4ce3b622a66f14c93bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:31.227550 21004 tablet_bootstrap.cc:492] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 99a0be6ef2144b7fb247ba9d3f051d66: Bootstrap starting.
I20240501 14:10:31.227656 21008 ts_tablet_manager.cc:1398] T 10aa2feeb0324aa4a6221d2f2f3fc941 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:31.228207 21003 tablet_bootstrap.cc:492] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367: No bootstrap required, opened a new log
I20240501 14:10:31.228308 21003 ts_tablet_manager.cc:1367] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:10:31.228410 21004 tablet_bootstrap.cc:654] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 99a0be6ef2144b7fb247ba9d3f051d66: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:31.228646 21003 raft_consensus.cc:348] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:31.228724 21003 raft_consensus.cc:374] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:31.228744 21003 raft_consensus.cc:729] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec8cde17a00645e7b38d5d29f7425367, State: Initialized, Role: FOLLOWER
I20240501 14:10:31.228842 21003 consensus_queue.cc:260] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:31.229101 21003 ts_tablet_manager.cc:1398] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:10:31.229295 21003 tablet_bootstrap.cc:492] T 7eea58d4a33f422a9ada1e82d408577f P ec8cde17a00645e7b38d5d29f7425367: Bootstrap starting.
I20240501 14:10:31.230199 21002 tablet_bootstrap.cc:492] T 7eea58d4a33f422a9ada1e82d408577f P da3c4e948ff747678f18456c134d7ed4: No bootstrap required, opened a new log
I20240501 14:10:31.230302 21003 tablet_bootstrap.cc:654] T 7eea58d4a33f422a9ada1e82d408577f P ec8cde17a00645e7b38d5d29f7425367: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:31.230296 21002 ts_tablet_manager.cc:1367] T 7eea58d4a33f422a9ada1e82d408577f P da3c4e948ff747678f18456c134d7ed4: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.002s
I20240501 14:10:31.230762 21002 raft_consensus.cc:348] T 7eea58d4a33f422a9ada1e82d408577f P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:31.230854 21002 raft_consensus.cc:374] T 7eea58d4a33f422a9ada1e82d408577f P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:31.230875 21002 raft_consensus.cc:729] T 7eea58d4a33f422a9ada1e82d408577f P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3c4e948ff747678f18456c134d7ed4, State: Initialized, Role: FOLLOWER
I20240501 14:10:31.230976 21002 consensus_queue.cc:260] T 7eea58d4a33f422a9ada1e82d408577f P da3c4e948ff747678f18456c134d7ed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:31.231258 21002 ts_tablet_manager.cc:1398] T 7eea58d4a33f422a9ada1e82d408577f P da3c4e948ff747678f18456c134d7ed4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:31.231740 21004 tablet_bootstrap.cc:492] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 99a0be6ef2144b7fb247ba9d3f051d66: No bootstrap required, opened a new log
I20240501 14:10:31.231843 21004 ts_tablet_manager.cc:1367] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240501 14:10:31.232364 21004 raft_consensus.cc:348] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:31.232457 21004 raft_consensus.cc:374] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:31.232496 21004 raft_consensus.cc:729] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a0be6ef2144b7fb247ba9d3f051d66, State: Initialized, Role: FOLLOWER
I20240501 14:10:31.232602 21003 tablet_bootstrap.cc:492] T 7eea58d4a33f422a9ada1e82d408577f P ec8cde17a00645e7b38d5d29f7425367: No bootstrap required, opened a new log
I20240501 14:10:31.232604 21004 consensus_queue.cc:260] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 99a0be6ef2144b7fb247ba9d3f051d66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:31.232679 21003 ts_tablet_manager.cc:1367] T 7eea58d4a33f422a9ada1e82d408577f P ec8cde17a00645e7b38d5d29f7425367: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:10:31.232854 21004 ts_tablet_manager.cc:1398] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:10:31.233139 21003 raft_consensus.cc:348] T 7eea58d4a33f422a9ada1e82d408577f P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:31.233254 21003 raft_consensus.cc:374] T 7eea58d4a33f422a9ada1e82d408577f P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:31.233289 21003 raft_consensus.cc:729] T 7eea58d4a33f422a9ada1e82d408577f P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec8cde17a00645e7b38d5d29f7425367, State: Initialized, Role: FOLLOWER
I20240501 14:10:31.233397 21003 consensus_queue.cc:260] T 7eea58d4a33f422a9ada1e82d408577f P ec8cde17a00645e7b38d5d29f7425367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:31.233652 21003 ts_tablet_manager.cc:1398] T 7eea58d4a33f422a9ada1e82d408577f P ec8cde17a00645e7b38d5d29f7425367: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:31.233801 21003 tablet_bootstrap.cc:492] T 10aa2feeb0324aa4a6221d2f2f3fc941 P ec8cde17a00645e7b38d5d29f7425367: Bootstrap starting.
I20240501 14:10:31.234654 21003 tablet_bootstrap.cc:654] T 10aa2feeb0324aa4a6221d2f2f3fc941 P ec8cde17a00645e7b38d5d29f7425367: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:31.235967 21003 tablet_bootstrap.cc:492] T 10aa2feeb0324aa4a6221d2f2f3fc941 P ec8cde17a00645e7b38d5d29f7425367: No bootstrap required, opened a new log
I20240501 14:10:31.236056 21003 ts_tablet_manager.cc:1367] T 10aa2feeb0324aa4a6221d2f2f3fc941 P ec8cde17a00645e7b38d5d29f7425367: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:10:31.236698 21003 raft_consensus.cc:348] T 10aa2feeb0324aa4a6221d2f2f3fc941 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:31.236800 21003 raft_consensus.cc:374] T 10aa2feeb0324aa4a6221d2f2f3fc941 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:31.236843 21003 raft_consensus.cc:729] T 10aa2feeb0324aa4a6221d2f2f3fc941 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec8cde17a00645e7b38d5d29f7425367, State: Initialized, Role: FOLLOWER
I20240501 14:10:31.236994 21003 consensus_queue.cc:260] T 10aa2feeb0324aa4a6221d2f2f3fc941 P ec8cde17a00645e7b38d5d29f7425367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:31.237269 21003 ts_tablet_manager.cc:1398] T 10aa2feeb0324aa4a6221d2f2f3fc941 P ec8cde17a00645e7b38d5d29f7425367: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:31.316931 20931 raft_consensus.cc:482] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:31.317067 20931 raft_consensus.cc:504] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:31.317503 20931 leader_election.cc:288] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829), da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357)
I20240501 14:10:31.317909 19768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe554c3145de46cf8070efcecffbaedb" candidate_uuid: "ec8cde17a00645e7b38d5d29f7425367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3c4e948ff747678f18456c134d7ed4" is_pre_election: true
I20240501 14:10:31.317926 19506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe554c3145de46cf8070efcecffbaedb" candidate_uuid: "ec8cde17a00645e7b38d5d29f7425367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" is_pre_election: true
I20240501 14:10:31.318058 19768 raft_consensus.cc:2454] T fe554c3145de46cf8070efcecffbaedb P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec8cde17a00645e7b38d5d29f7425367 in term 0.
I20240501 14:10:31.318075 19506 raft_consensus.cc:2454] T fe554c3145de46cf8070efcecffbaedb P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec8cde17a00645e7b38d5d29f7425367 in term 0.
I20240501 14:10:31.318336 19573 leader_election.cc:302] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da3c4e948ff747678f18456c134d7ed4, ec8cde17a00645e7b38d5d29f7425367; no voters:
I20240501 14:10:31.318499 20931 raft_consensus.cc:2781] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:31.318554 20931 raft_consensus.cc:482] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:31.318598 20931 raft_consensus.cc:3037] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:31.319504 20931 raft_consensus.cc:504] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:31.319799 20931 leader_election.cc:288] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 [CANDIDATE]: Term 1 election: Requested vote from peers 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829), da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357)
I20240501 14:10:31.320116 19506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe554c3145de46cf8070efcecffbaedb" candidate_uuid: "ec8cde17a00645e7b38d5d29f7425367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a0be6ef2144b7fb247ba9d3f051d66"
I20240501 14:10:31.320201 19768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe554c3145de46cf8070efcecffbaedb" candidate_uuid: "ec8cde17a00645e7b38d5d29f7425367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3c4e948ff747678f18456c134d7ed4"
I20240501 14:10:31.320250 19506 raft_consensus.cc:3037] T fe554c3145de46cf8070efcecffbaedb P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:31.320415 19768 raft_consensus.cc:3037] T fe554c3145de46cf8070efcecffbaedb P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:31.321017 19506 raft_consensus.cc:2454] T fe554c3145de46cf8070efcecffbaedb P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec8cde17a00645e7b38d5d29f7425367 in term 1.
I20240501 14:10:31.321178 19768 raft_consensus.cc:2454] T fe554c3145de46cf8070efcecffbaedb P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec8cde17a00645e7b38d5d29f7425367 in term 1.
I20240501 14:10:31.321270 19572 leader_election.cc:302] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99a0be6ef2144b7fb247ba9d3f051d66, ec8cde17a00645e7b38d5d29f7425367; no voters:
I20240501 14:10:31.321429 20931 raft_consensus.cc:2781] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:31.321547 20931 raft_consensus.cc:686] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Becoming Leader. State: Replica: ec8cde17a00645e7b38d5d29f7425367, State: Running, Role: LEADER
I20240501 14:10:31.321682 20931 consensus_queue.cc:237] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:31.323371 19225 catalog_manager.cc:5484] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131). New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } }
I20240501 14:10:31.336608 20931 raft_consensus.cc:482] T 10aa2feeb0324aa4a6221d2f2f3fc941 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:31.336743 20931 raft_consensus.cc:504] T 10aa2feeb0324aa4a6221d2f2f3fc941 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:31.337200 20931 leader_election.cc:288] T 10aa2feeb0324aa4a6221d2f2f3fc941 P ec8cde17a00645e7b38d5d29f7425367 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829), fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847)
I20240501 14:10:31.337646 19506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10aa2feeb0324aa4a6221d2f2f3fc941" candidate_uuid: "ec8cde17a00645e7b38d5d29f7425367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" is_pre_election: true
I20240501 14:10:31.337633 19375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10aa2feeb0324aa4a6221d2f2f3fc941" candidate_uuid: "ec8cde17a00645e7b38d5d29f7425367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" is_pre_election: true
I20240501 14:10:31.337782 19375 raft_consensus.cc:2454] T 10aa2feeb0324aa4a6221d2f2f3fc941 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec8cde17a00645e7b38d5d29f7425367 in term 0.
I20240501 14:10:31.337795 19506 raft_consensus.cc:2454] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec8cde17a00645e7b38d5d29f7425367 in term 0.
I20240501 14:10:31.338099 19572 leader_election.cc:302] T 10aa2feeb0324aa4a6221d2f2f3fc941 P ec8cde17a00645e7b38d5d29f7425367 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99a0be6ef2144b7fb247ba9d3f051d66, ec8cde17a00645e7b38d5d29f7425367; no voters:
I20240501 14:10:31.338285 20931 raft_consensus.cc:2781] T 10aa2feeb0324aa4a6221d2f2f3fc941 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:31.338346 20931 raft_consensus.cc:482] T 10aa2feeb0324aa4a6221d2f2f3fc941 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:31.338383 20931 raft_consensus.cc:3037] T 10aa2feeb0324aa4a6221d2f2f3fc941 P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:31.339344 20931 raft_consensus.cc:504] T 10aa2feeb0324aa4a6221d2f2f3fc941 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:31.339650 20931 leader_election.cc:288] T 10aa2feeb0324aa4a6221d2f2f3fc941 P ec8cde17a00645e7b38d5d29f7425367 [CANDIDATE]: Term 1 election: Requested vote from peers 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829), fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847)
I20240501 14:10:31.339926 19375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10aa2feeb0324aa4a6221d2f2f3fc941" candidate_uuid: "ec8cde17a00645e7b38d5d29f7425367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc"
I20240501 14:10:31.339926 19506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10aa2feeb0324aa4a6221d2f2f3fc941" candidate_uuid: "ec8cde17a00645e7b38d5d29f7425367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a0be6ef2144b7fb247ba9d3f051d66"
I20240501 14:10:31.340035 19506 raft_consensus.cc:3037] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:31.340045 19375 raft_consensus.cc:3037] T 10aa2feeb0324aa4a6221d2f2f3fc941 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:31.340965 19506 raft_consensus.cc:2454] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec8cde17a00645e7b38d5d29f7425367 in term 1.
I20240501 14:10:31.340965 19375 raft_consensus.cc:2454] T 10aa2feeb0324aa4a6221d2f2f3fc941 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec8cde17a00645e7b38d5d29f7425367 in term 1.
I20240501 14:10:31.341298 19571 leader_election.cc:302] T 10aa2feeb0324aa4a6221d2f2f3fc941 P ec8cde17a00645e7b38d5d29f7425367 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ec8cde17a00645e7b38d5d29f7425367, fb1e6bc2d07b4ce3b622a66f14c93bdc; no voters:
I20240501 14:10:31.341455 20931 raft_consensus.cc:2781] T 10aa2feeb0324aa4a6221d2f2f3fc941 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:31.341555 20931 raft_consensus.cc:686] T 10aa2feeb0324aa4a6221d2f2f3fc941 P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Becoming Leader. State: Replica: ec8cde17a00645e7b38d5d29f7425367, State: Running, Role: LEADER
I20240501 14:10:31.341706 20931 consensus_queue.cc:237] T 10aa2feeb0324aa4a6221d2f2f3fc941 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:10:31.343502 19225 catalog_manager.cc:5484] T 10aa2feeb0324aa4a6221d2f2f3fc941 P ec8cde17a00645e7b38d5d29f7425367 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131). New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } }
I20240501 14:10:31.346518 20933 raft_consensus.cc:482] T 7eea58d4a33f422a9ada1e82d408577f P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:31.346628 20933 raft_consensus.cc:504] T 7eea58d4a33f422a9ada1e82d408577f P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:31.346999 20933 leader_election.cc:288] T 7eea58d4a33f422a9ada1e82d408577f P da3c4e948ff747678f18456c134d7ed4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829), ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201)
I20240501 14:10:31.347342 19506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eea58d4a33f422a9ada1e82d408577f" candidate_uuid: "da3c4e948ff747678f18456c134d7ed4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" is_pre_election: true
I20240501 14:10:31.347457 19506 raft_consensus.cc:2454] T 7eea58d4a33f422a9ada1e82d408577f P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da3c4e948ff747678f18456c134d7ed4 in term 0.
I20240501 14:10:31.347420 19637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eea58d4a33f422a9ada1e82d408577f" candidate_uuid: "da3c4e948ff747678f18456c134d7ed4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec8cde17a00645e7b38d5d29f7425367" is_pre_election: true
I20240501 14:10:31.347550 19637 raft_consensus.cc:2454] T 7eea58d4a33f422a9ada1e82d408577f P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da3c4e948ff747678f18456c134d7ed4 in term 0.
I20240501 14:10:31.347764 19703 leader_election.cc:302] T 7eea58d4a33f422a9ada1e82d408577f P da3c4e948ff747678f18456c134d7ed4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99a0be6ef2144b7fb247ba9d3f051d66, da3c4e948ff747678f18456c134d7ed4; no voters:
I20240501 14:10:31.347949 20933 raft_consensus.cc:2781] T 7eea58d4a33f422a9ada1e82d408577f P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:31.348013 20933 raft_consensus.cc:482] T 7eea58d4a33f422a9ada1e82d408577f P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:31.348045 20933 raft_consensus.cc:3037] T 7eea58d4a33f422a9ada1e82d408577f P da3c4e948ff747678f18456c134d7ed4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:31.349076 20933 raft_consensus.cc:504] T 7eea58d4a33f422a9ada1e82d408577f P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:31.349380 20933 leader_election.cc:288] T 7eea58d4a33f422a9ada1e82d408577f P da3c4e948ff747678f18456c134d7ed4 [CANDIDATE]: Term 1 election: Requested vote from peers 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829), ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201)
I20240501 14:10:31.349692 19506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eea58d4a33f422a9ada1e82d408577f" candidate_uuid: "da3c4e948ff747678f18456c134d7ed4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a0be6ef2144b7fb247ba9d3f051d66"
I20240501 14:10:31.349740 19637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eea58d4a33f422a9ada1e82d408577f" candidate_uuid: "da3c4e948ff747678f18456c134d7ed4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec8cde17a00645e7b38d5d29f7425367"
I20240501 14:10:31.349817 19506 raft_consensus.cc:3037] T 7eea58d4a33f422a9ada1e82d408577f P 99a0be6ef2144b7fb247ba9d3f051d66 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:31.349845 19637 raft_consensus.cc:3037] T 7eea58d4a33f422a9ada1e82d408577f P ec8cde17a00645e7b38d5d29f7425367 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:31.350755 19637 raft_consensus.cc:2454] T 7eea58d4a33f422a9ada1e82d408577f P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da3c4e948ff747678f18456c134d7ed4 in term 1.
I20240501 14:10:31.350755 19506 raft_consensus.cc:2454] T 7eea58d4a33f422a9ada1e82d408577f P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da3c4e948ff747678f18456c134d7ed4 in term 1.
I20240501 14:10:31.351022 19703 leader_election.cc:302] T 7eea58d4a33f422a9ada1e82d408577f P da3c4e948ff747678f18456c134d7ed4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da3c4e948ff747678f18456c134d7ed4, ec8cde17a00645e7b38d5d29f7425367; no voters:
I20240501 14:10:31.351228 20933 raft_consensus.cc:2781] T 7eea58d4a33f422a9ada1e82d408577f P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:31.351318 20933 raft_consensus.cc:686] T 7eea58d4a33f422a9ada1e82d408577f P da3c4e948ff747678f18456c134d7ed4 [term 1 LEADER]: Becoming Leader. State: Replica: da3c4e948ff747678f18456c134d7ed4, State: Running, Role: LEADER
I20240501 14:10:31.351457 20933 consensus_queue.cc:237] T 7eea58d4a33f422a9ada1e82d408577f P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:31.353309 19225 catalog_manager.cc:5484] T 7eea58d4a33f422a9ada1e82d408577f P da3c4e948ff747678f18456c134d7ed4 reported cstate change: term changed from 0 to 1, leader changed from <none> to da3c4e948ff747678f18456c134d7ed4 (127.7.39.132). New cstate: current_term: 1 leader_uuid: "da3c4e948ff747678f18456c134d7ed4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } }
W20240501 14:10:31.485879 21018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:10:31.486116 21018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:10:31.488981 21018 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240501 14:10:31.566965 19506 raft_consensus.cc:1261] T 7eea58d4a33f422a9ada1e82d408577f P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:31.567915 19376 raft_consensus.cc:1261] T 10aa2feeb0324aa4a6221d2f2f3fc941 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:31.568011 19768 raft_consensus.cc:1261] T fe554c3145de46cf8070efcecffbaedb P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:31.568382 19505 raft_consensus.cc:1261] T fe554c3145de46cf8070efcecffbaedb P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:31.568464 20859 consensus_queue.cc:1035] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:31.568629 20654 consensus_queue.cc:1035] T 10aa2feeb0324aa4a6221d2f2f3fc941 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:31.568698 19507 raft_consensus.cc:1261] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:31.569139 20654 consensus_queue.cc:1035] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:31.569417 20654 consensus_queue.cc:1035] T 10aa2feeb0324aa4a6221d2f2f3fc941 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:31.571120 19638 raft_consensus.cc:1261] T 7eea58d4a33f422a9ada1e82d408577f P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:31.571125 20933 consensus_queue.cc:1035] T 7eea58d4a33f422a9ada1e82d408577f P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:31.586180 20831 consensus_queue.cc:1035] T 7eea58d4a33f422a9ada1e82d408577f P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:31.590106 20972 mvcc.cc:205] Tried to move back new op lower bound from 7022889498884497408 to 7022889498017120256. Current Snapshot: MvccSnapshot[applied={T|T < 7022889498884497408}]
INSERT report
rows total: 2000
time total: 117.848 ms
time per row: 0.0589239 ms
I20240501 14:10:31.644641 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.133:46883
--local_ip_for_outbound_sockets=127.7.39.133
--tserver_master_addrs=127.7.39.190:34571
--webserver_port=33707
--webserver_interface=127.7.39.133
--builtin_ntp_servers=127.7.39.148:46161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 14:10:31.753453 21055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:10:31.753666 21055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:10:31.753723 21055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:10:31.756815 21055 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:10:31.756888 21055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:10:31.756970 21055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.133
I20240501 14:10:31.760313 21055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:46161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.133:46883
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.7.39.133
--webserver_port=33707
--tserver_master_addrs=127.7.39.190:34571
--never_fsync=true
--heap_profile_path=/tmp/kudu.21055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.133
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:10:31.761139 21055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:10:31.762151 21055 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 14:10:31.771296 21055 server_base.cc:1034] running on GCE node
W20240501 14:10:31.772181 21061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:10:31.772754 21063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:10:31.773909 21060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:10:31.774293 21055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:10:31.774782 21055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:10:31.775931 21055 hybrid_clock.cc:648] HybridClock initialized: now 1714572631775919 us; error 25 us; skew 500 ppm
I20240501 14:10:31.777735 21055 webserver.cc:460] Webserver started at http://127.7.39.133:33707/ using document root <none> and password file <none>
I20240501 14:10:31.778266 21055 fs_manager.cc:359] Metadata directory not provided
I20240501 14:10:31.778352 21055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:10:31.781388 21055 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:10:31.783056 21069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:31.783948 21055 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:10:31.784066 21055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "5bf81f3d986e428190d3f7727f36d96c"
format_stamp: "Formatted at 2024-05-01 14:10:21 on dist-test-slave-rnfw"
I20240501 14:10:31.784472 21055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:10:31.796761 21055 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:10:31.797291 21055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:10:31.797446 21055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:10:31.797901 21055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:10:31.798732 21055 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:10:31.798800 21055 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:31.798844 21055 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:10:31.798879 21055 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:31.806532 21055 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.133:46883
I20240501 14:10:31.806572 21182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.133:46883 every 8 connection(s)
I20240501 14:10:31.807473 21055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 14:10:31.810441 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 21055
I20240501 14:10:31.810833 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.134:41911
--local_ip_for_outbound_sockets=127.7.39.134
--tserver_master_addrs=127.7.39.190:34571
--webserver_port=42995
--webserver_interface=127.7.39.134
--builtin_ntp_servers=127.7.39.148:46161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:10:31.816711 21183 heartbeater.cc:347] Connected to a master server at 127.7.39.190:34571
I20240501 14:10:31.817167 21183 heartbeater.cc:464] Registering TS with master...
I20240501 14:10:31.817831 21183 heartbeater.cc:511] Master 127.7.39.190:34571 requested a full tablet report, sending...
I20240501 14:10:31.819029 19233 ts_manager.cc:194] Re-registered known tserver with Master: 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133:46883)
I20240501 14:10:31.819636 19233 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.133:48905
W20240501 14:10:31.920102 21186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:10:31.920312 21186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:10:31.920353 21186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:10:31.923259 21186 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:10:31.923317 21186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:10:31.923398 21186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.134
I20240501 14:10:31.926961 21186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:46161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.134:41911
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.7.39.134
--webserver_port=42995
--tserver_master_addrs=127.7.39.190:34571
--never_fsync=true
--heap_profile_path=/tmp/kudu.21186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.134
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:10:31.927820 21186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:10:31.928886 21186 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:10:31.937844 21192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:10:31.939112 21194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:10:31.939664 21191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:10:31.940783 21186 server_base.cc:1034] running on GCE node
I20240501 14:10:31.941160 21186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:10:31.941704 21186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:10:31.942945 21186 hybrid_clock.cc:648] HybridClock initialized: now 1714572631942922 us; error 36 us; skew 500 ppm
I20240501 14:10:31.944931 21186 webserver.cc:460] Webserver started at http://127.7.39.134:42995/ using document root <none> and password file <none>
I20240501 14:10:31.945475 21186 fs_manager.cc:359] Metadata directory not provided
I20240501 14:10:31.945554 21186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:10:31.948740 21186 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:10:31.950493 21200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:31.951506 21186 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:10:31.951601 21186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "e6246b183293412494919f1e7efc9448"
format_stamp: "Formatted at 2024-05-01 14:10:22 on dist-test-slave-rnfw"
I20240501 14:10:31.951964 21186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:10:31.968565 21186 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:10:31.969131 21186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:10:31.969274 21186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:10:31.969755 21186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:10:31.970626 21186 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:10:31.970685 21186 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:31.970731 21186 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:10:31.970772 21186 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:31.978945 21186 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.134:41911
I20240501 14:10:31.978999 21313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.134:41911 every 8 connection(s)
I20240501 14:10:31.979907 21186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 14:10:31.987707 21314 heartbeater.cc:347] Connected to a master server at 127.7.39.190:34571
I20240501 14:10:31.987972 21314 heartbeater.cc:464] Registering TS with master...
I20240501 14:10:31.988046 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 21186
I20240501 14:10:31.988406 21314 heartbeater.cc:511] Master 127.7.39.190:34571 requested a full tablet report, sending...
I20240501 14:10:31.988487 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.135:38751
--local_ip_for_outbound_sockets=127.7.39.135
--tserver_master_addrs=127.7.39.190:34571
--webserver_port=33933
--webserver_interface=127.7.39.135
--builtin_ntp_servers=127.7.39.148:46161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:10:31.989428 19233 ts_manager.cc:194] Re-registered known tserver with Master: e6246b183293412494919f1e7efc9448 (127.7.39.134:41911)
I20240501 14:10:31.990183 19233 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.134:35761
W20240501 14:10:32.096927 21317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:10:32.097162 21317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:10:32.097225 21317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:10:32.100431 21317 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:10:32.100500 21317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:10:32.100582 21317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.135
I20240501 14:10:32.104238 21317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:46161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.135:38751
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.7.39.135
--webserver_port=33933
--tserver_master_addrs=127.7.39.190:34571
--never_fsync=true
--heap_profile_path=/tmp/kudu.21317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.135
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:10:32.105068 21317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:10:32.106141 21317 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:10:32.114023 21322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:10:32.115396 21323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:10:32.116717 21317 server_base.cc:1034] running on GCE node
W20240501 14:10:32.117842 21325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:10:32.118233 21317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:10:32.118791 21317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:10:32.119935 21317 hybrid_clock.cc:648] HybridClock initialized: now 1714572632119922 us; error 28 us; skew 500 ppm
I20240501 14:10:32.121749 21317 webserver.cc:460] Webserver started at http://127.7.39.135:33933/ using document root <none> and password file <none>
I20240501 14:10:32.122280 21317 fs_manager.cc:359] Metadata directory not provided
I20240501 14:10:32.122350 21317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:10:32.125591 21317 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:10:32.127481 21331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:32.128516 21317 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:10:32.128618 21317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "43ce4a4c954f4a5f80954b38933c189d"
format_stamp: "Formatted at 2024-05-01 14:10:22 on dist-test-slave-rnfw"
I20240501 14:10:32.129070 21317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:10:32.156289 21317 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:10:32.156864 21317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:10:32.157027 21317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:10:32.157545 21317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:10:32.158367 21317 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:10:32.158430 21317 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:32.158483 21317 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:10:32.158506 21317 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:32.166690 21317 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.135:38751
I20240501 14:10:32.166721 21444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.135:38751 every 8 connection(s)
I20240501 14:10:32.167683 21317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 14:10:32.175797 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 21317
I20240501 14:10:32.176296 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.136:34895
--local_ip_for_outbound_sockets=127.7.39.136
--tserver_master_addrs=127.7.39.190:34571
--webserver_port=42151
--webserver_interface=127.7.39.136
--builtin_ntp_servers=127.7.39.148:46161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:10:32.177294 21445 heartbeater.cc:347] Connected to a master server at 127.7.39.190:34571
I20240501 14:10:32.177489 21445 heartbeater.cc:464] Registering TS with master...
I20240501 14:10:32.177958 21445 heartbeater.cc:511] Master 127.7.39.190:34571 requested a full tablet report, sending...
I20240501 14:10:32.178977 19233 ts_manager.cc:194] Re-registered known tserver with Master: 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135:38751)
I20240501 14:10:32.179620 19233 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.135:44145
W20240501 14:10:32.292405 21448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:10:32.292626 21448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:10:32.292677 21448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:10:32.295550 21448 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:10:32.295609 21448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:10:32.295701 21448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.136
I20240501 14:10:32.299144 21448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:46161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.136:34895
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.7.39.136
--webserver_port=42151
--tserver_master_addrs=127.7.39.190:34571
--never_fsync=true
--heap_profile_path=/tmp/kudu.21448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.136
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:10:32.300000 21448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:10:32.301039 21448 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:10:32.310288 21456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:10:32.310732 21453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:10:32.312106 21454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:10:32.313154 21448 server_base.cc:1034] running on GCE node
I20240501 14:10:32.313519 21448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:10:32.314056 21448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:10:32.315199 21448 hybrid_clock.cc:648] HybridClock initialized: now 1714572632315186 us; error 26 us; skew 500 ppm
I20240501 14:10:32.320925 21448 webserver.cc:460] Webserver started at http://127.7.39.136:42151/ using document root <none> and password file <none>
I20240501 14:10:32.321485 21448 fs_manager.cc:359] Metadata directory not provided
I20240501 14:10:32.321558 21448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:10:32.324769 21448 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:10:32.326736 21462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:32.327672 21448 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:10:32.327770 21448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "778c78105092428bbcde75ab2cdb6a05"
format_stamp: "Formatted at 2024-05-01 14:10:22 on dist-test-slave-rnfw"
I20240501 14:10:32.328197 21448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:10:32.343083 21448 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:10:32.343655 21448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:10:32.343806 21448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:10:32.344329 21448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:10:32.345189 21448 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:10:32.345237 21448 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:32.345294 21448 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:10:32.345319 21448 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:10:32.353349 21448 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.136:34895
I20240501 14:10:32.353374 21575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.136:34895 every 8 connection(s)
I20240501 14:10:32.354328 21448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 14:10:32.357486 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 21448
I20240501 14:10:32.364931 21576 heartbeater.cc:347] Connected to a master server at 127.7.39.190:34571
I20240501 14:10:32.365151 21576 heartbeater.cc:464] Registering TS with master...
I20240501 14:10:32.365742 21576 heartbeater.cc:511] Master 127.7.39.190:34571 requested a full tablet report, sending...
I20240501 14:10:32.366825 19233 ts_manager.cc:194] Re-registered known tserver with Master: 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136:34895)
I20240501 14:10:32.367597 19233 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.136:55501
I20240501 14:10:32.751309 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:32.752066 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:32.760569 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:32.785609 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:32.790668 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:32.791291 21379 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:32.796149 21248 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:32.821890 21183 heartbeater.cc:503] Master 127.7.39.190:34571 was elected leader, sending a full tablet report...
I20240501 14:10:32.827847 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:32.828433 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:32.831272 21248 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:32.845235 19618 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:32.860183 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:32.860993 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:32.863497 21379 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:32.873911 21379 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:32.874637 19618 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:32.885110 19618 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:32.901182 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:32.902570 21248 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:32.908103 21379 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:32.910285 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:32.924679 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:32.929903 21379 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:32.936434 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:32.944000 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:32.947628 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:32.949568 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:32.952411 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:32.955404 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:32.956574 21248 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:32.961993 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:32.963340 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:32.968400 19618 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:32.971758 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:32.971975 19749 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:32.972174 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:32.990644 21248 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:32.994856 21314 heartbeater.cc:503] Master 127.7.39.190:34571 was elected leader, sending a full tablet report...
I20240501 14:10:32.999214 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:33.056188 19618 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:33.058498 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:33.182102 21445 heartbeater.cc:503] Master 127.7.39.190:34571 was elected leader, sending a full tablet report...
I20240501 14:10:33.311707 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:33.318420 21379 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:33.319355 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:33.327227 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:33.332607 21379 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:33.337327 21248 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:33.341938 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:33.348416 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:33.359851 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:33.369565 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:33.370154 21576 heartbeater.cc:503] Master 127.7.39.190:34571 was elected leader, sending a full tablet report...
I20240501 14:10:33.371235 19618 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:33.372617 21379 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:33.374050 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:33.380601 21248 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:33.382767 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:33.384593 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:33.395622 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:33.398736 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:33.413034 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:33.427673 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:33.429630 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:33.453877 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:33.459604 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:33.461364 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:33.468904 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:33.471840 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:33.481827 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:33.482616 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:33.494752 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:33.500627 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:33.503167 19618 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:33.511793 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:33.528856 21248 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:33.533428 21248 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:33.536852 21248 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:33.539402 19618 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:33.540272 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:33.540336 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:33.546391 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:33.577180 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:33.785001 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:33.785825 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:33.799595 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:33.821519 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:33.821780 21248 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:33.827898 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:33.834184 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:33.857834 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:33.870862 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:33.874313 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:33.880084 21248 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:33.886922 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:33.887847 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:33.892851 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:33.895512 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:33.908232 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:33.945847 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:33.948565 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:33.961846 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:33.968284 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:33.973802 21248 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:33.975476 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:33.975574 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:33.993687 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:34.002048 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:34.003192 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:34.008772 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:34.018957 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:34.034399 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:34.048516 21248 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:34.056887 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:34.074707 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:34.079825 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:34.080812 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:34.088496 21248 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:34.088618 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:34.091351 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:34.103121 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:34.106968 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:34.109867 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:34.185353 19506 consensus_queue.cc:237] T b6be9a115fe14c098c29cad819849f95 P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:34.187268 19768 raft_consensus.cc:1261] T b6be9a115fe14c098c29cad819849f95 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Refusing update from remote peer 99a0be6ef2144b7fb247ba9d3f051d66: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:34.188702 19637 raft_consensus.cc:1261] T b6be9a115fe14c098c29cad819849f95 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Refusing update from remote peer 99a0be6ef2144b7fb247ba9d3f051d66: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:34.189036 22081 consensus_queue.cc:1035] T b6be9a115fe14c098c29cad819849f95 P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:34.189802 22081 consensus_queue.cc:1035] T b6be9a115fe14c098c29cad819849f95 P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:34.191835 22081 raft_consensus.cc:2932] T b6be9a115fe14c098c29cad819849f95 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:34.194192 19768 raft_consensus.cc:2932] T b6be9a115fe14c098c29cad819849f95 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:34.196698 19637 raft_consensus.cc:2932] T b6be9a115fe14c098c29cad819849f95 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
W20240501 14:10:34.197659 19440 consensus_peers.cc:485] T b6be9a115fe14c098c29cad819849f95 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135:38751): Couldn't send request to peer 43ce4a4c954f4a5f80954b38933c189d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b6be9a115fe14c098c29cad819849f95. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:34.199730 19232 catalog_manager.cc:5484] T b6be9a115fe14c098c29cad819849f95 P 99a0be6ef2144b7fb247ba9d3f051d66 reported cstate change: config changed from index -1 to 4, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New cstate: current_term: 1 leader_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:34.213943 19768 consensus_queue.cc:237] T cf62635e9a994259b9e293e5eb425b55 P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:10:34.217494 19637 raft_consensus.cc:1261] T cf62635e9a994259b9e293e5eb425b55 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:34.217523 19506 raft_consensus.cc:1261] T cf62635e9a994259b9e293e5eb425b55 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:34.218016 20933 consensus_queue.cc:1035] T cf62635e9a994259b9e293e5eb425b55 P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:34.219583 20933 consensus_queue.cc:1035] T cf62635e9a994259b9e293e5eb425b55 P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:34.222950 22410 raft_consensus.cc:2932] T cf62635e9a994259b9e293e5eb425b55 P da3c4e948ff747678f18456c134d7ed4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:34.223593 19637 raft_consensus.cc:2932] T cf62635e9a994259b9e293e5eb425b55 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:34.224916 19768 consensus_queue.cc:237] T 7eea58d4a33f422a9ada1e82d408577f P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:34.230005 19506 raft_consensus.cc:1261] T 7eea58d4a33f422a9ada1e82d408577f P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:34.230161 19637 raft_consensus.cc:1261] T 7eea58d4a33f422a9ada1e82d408577f P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:34.229715 19228 catalog_manager.cc:5484] T cf62635e9a994259b9e293e5eb425b55 P ec8cde17a00645e7b38d5d29f7425367 reported cstate change: config changed from index -1 to 4, VOTER 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136) added. New cstate: current_term: 1 leader_uuid: "da3c4e948ff747678f18456c134d7ed4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:34.231608 22411 consensus_queue.cc:1035] T 7eea58d4a33f422a9ada1e82d408577f P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:34.232802 22406 consensus_queue.cc:1035] T 7eea58d4a33f422a9ada1e82d408577f P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:34.235664 19506 raft_consensus.cc:2932] T cf62635e9a994259b9e293e5eb425b55 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
W20240501 14:10:34.236596 19702 consensus_peers.cc:485] T 7eea58d4a33f422a9ada1e82d408577f P da3c4e948ff747678f18456c134d7ed4 -> Peer 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135:38751): Couldn't send request to peer 43ce4a4c954f4a5f80954b38933c189d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7eea58d4a33f422a9ada1e82d408577f. This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:34.237160 19702 consensus_peers.cc:485] T cf62635e9a994259b9e293e5eb425b55 P da3c4e948ff747678f18456c134d7ed4 -> Peer 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136:34895): Couldn't send request to peer 778c78105092428bbcde75ab2cdb6a05. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cf62635e9a994259b9e293e5eb425b55. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:34.239914 19637 raft_consensus.cc:2932] T 7eea58d4a33f422a9ada1e82d408577f P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:34.240456 22411 raft_consensus.cc:2932] T 7eea58d4a33f422a9ada1e82d408577f P da3c4e948ff747678f18456c134d7ed4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:34.243220 19506 raft_consensus.cc:2932] T 7eea58d4a33f422a9ada1e82d408577f P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:34.243717 19228 catalog_manager.cc:5484] T 7eea58d4a33f422a9ada1e82d408577f P ec8cde17a00645e7b38d5d29f7425367 reported cstate change: config changed from index -1 to 4, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New cstate: current_term: 1 leader_uuid: "da3c4e948ff747678f18456c134d7ed4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:34.247910 19637 consensus_queue.cc:237] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:34.249675 19768 raft_consensus.cc:1261] T db0f01ee72b24ec9b84410cf94ef9759 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:34.250149 20654 consensus_queue.cc:1035] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:34.250177 19375 raft_consensus.cc:1261] T db0f01ee72b24ec9b84410cf94ef9759 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:34.250587 20654 consensus_queue.cc:1035] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:34.252806 19637 consensus_queue.cc:237] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:34.254554 19768 raft_consensus.cc:1261] T 85593473500b4d5eaf9f8b0bf2296777 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:34.254956 22086 consensus_queue.cc:1035] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:34.255316 19375 raft_consensus.cc:1261] T 85593473500b4d5eaf9f8b0bf2296777 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:34.255515 22086 raft_consensus.cc:2932] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:34.257649 22458 consensus_queue.cc:1035] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:34.257534 19375 raft_consensus.cc:2932] T db0f01ee72b24ec9b84410cf94ef9759 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:34.258926 19768 raft_consensus.cc:2932] T db0f01ee72b24ec9b84410cf94ef9759 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:34.261735 19228 catalog_manager.cc:5484] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 reported cstate change: config changed from index -1 to 4, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:34.264765 22086 raft_consensus.cc:2932] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:34.265208 19375 raft_consensus.cc:2932] T 85593473500b4d5eaf9f8b0bf2296777 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
W20240501 14:10:34.268793 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 85593473500b4d5eaf9f8b0bf2296777. This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:34.268992 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: db0f01ee72b24ec9b84410cf94ef9759. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:34.270627 19769 raft_consensus.cc:2932] T 85593473500b4d5eaf9f8b0bf2296777 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:34.272320 19226 catalog_manager.cc:5484] T 85593473500b4d5eaf9f8b0bf2296777 P fb1e6bc2d07b4ce3b622a66f14c93bdc reported cstate change: config changed from index -1 to 4, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:34.289629 19769 consensus_queue.cc:237] T 7eea58d4a33f422a9ada1e82d408577f P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:34.291499 19506 raft_consensus.cc:1261] T 7eea58d4a33f422a9ada1e82d408577f P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:34.292125 22250 consensus_queue.cc:1035] T 7eea58d4a33f422a9ada1e82d408577f P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:34.292258 19637 raft_consensus.cc:1261] T 7eea58d4a33f422a9ada1e82d408577f P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 14:10:34.292986 19702 consensus_peers.cc:485] T 7eea58d4a33f422a9ada1e82d408577f P da3c4e948ff747678f18456c134d7ed4 -> Peer 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135:38751): Couldn't send request to peer 43ce4a4c954f4a5f80954b38933c189d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7eea58d4a33f422a9ada1e82d408577f. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:34.294132 22250 consensus_queue.cc:1035] T 7eea58d4a33f422a9ada1e82d408577f P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:34.297008 22250 raft_consensus.cc:2932] T 7eea58d4a33f422a9ada1e82d408577f P da3c4e948ff747678f18456c134d7ed4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
W20240501 14:10:34.297768 19705 consensus_peers.cc:485] T 7eea58d4a33f422a9ada1e82d408577f P da3c4e948ff747678f18456c134d7ed4 -> Peer 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133:46883): Couldn't send request to peer 5bf81f3d986e428190d3f7727f36d96c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7eea58d4a33f422a9ada1e82d408577f. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:34.299300 19506 raft_consensus.cc:2932] T 7eea58d4a33f422a9ada1e82d408577f P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:34.301431 19226 catalog_manager.cc:5484] T 7eea58d4a33f422a9ada1e82d408577f P da3c4e948ff747678f18456c134d7ed4 reported cstate change: config changed from index 4 to 5, VOTER 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133) added. New cstate: current_term: 1 leader_uuid: "da3c4e948ff747678f18456c134d7ed4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:34.301529 19637 raft_consensus.cc:2932] T 7eea58d4a33f422a9ada1e82d408577f P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:34.312750 19375 consensus_queue.cc:237] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:34.312835 19376 consensus_queue.cc:237] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:34.314838 19506 raft_consensus.cc:1261] T ad7c255baee74e629e2976a942a4174f P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:34.316210 19506 raft_consensus.cc:1261] T f69020bb6c244848a6b0a50fd2b4f3eb P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:34.316785 22237 consensus_queue.cc:1035] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:34.317138 19637 raft_consensus.cc:1261] T f69020bb6c244848a6b0a50fd2b4f3eb P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:34.317569 21048 consensus_queue.cc:1035] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:34.319903 22528 consensus_queue.cc:1035] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:34.320737 19769 raft_consensus.cc:1261] T ad7c255baee74e629e2976a942a4174f P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:34.321272 22237 consensus_queue.cc:1035] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:34.322980 19376 consensus_queue.cc:237] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
W20240501 14:10:34.324136 19309 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135:38751): Couldn't send request to peer 43ce4a4c954f4a5f80954b38933c189d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ad7c255baee74e629e2976a942a4174f. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:34.324844 19638 raft_consensus.cc:1261] T 98af8b06b9324e3fa7cdb898b7b9fab9 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:10:34.325060 19309 consensus_peers.cc:485] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135:38751): Couldn't send request to peer 43ce4a4c954f4a5f80954b38933c189d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 98af8b06b9324e3fa7cdb898b7b9fab9. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:34.325399 22528 consensus_queue.cc:1035] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:34.326556 19507 raft_consensus.cc:1261] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:34.327827 22528 consensus_queue.cc:1035] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:10:34.331449 19312 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133:46883): Couldn't send request to peer 5bf81f3d986e428190d3f7727f36d96c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f69020bb6c244848a6b0a50fd2b4f3eb. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:34.331998 22524 raft_consensus.cc:2932] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:34.335013 19638 raft_consensus.cc:2932] T 98af8b06b9324e3fa7cdb898b7b9fab9 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:34.335402 19226 catalog_manager.cc:5484] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc reported cstate change: config changed from index -1 to 4, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New cstate: current_term: 1 leader_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:34.338491 19507 raft_consensus.cc:2932] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:34.342733 22237 raft_consensus.cc:2932] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:34.343122 21048 raft_consensus.cc:2932] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:34.345216 19769 raft_consensus.cc:2932] T ad7c255baee74e629e2976a942a4174f P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:34.345613 19637 raft_consensus.cc:2932] T f69020bb6c244848a6b0a50fd2b4f3eb P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:34.347690 19506 raft_consensus.cc:2932] T ad7c255baee74e629e2976a942a4174f P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:34.348709 19232 catalog_manager.cc:5484] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc reported cstate change: config changed from index -1 to 4, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New cstate: current_term: 1 leader_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:34.348871 19507 raft_consensus.cc:2932] T f69020bb6c244848a6b0a50fd2b4f3eb P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:34.348968 19232 catalog_manager.cc:5484] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc reported cstate change: config changed from index -1 to 4, VOTER 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133) added. New cstate: current_term: 1 leader_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:34.367594 19637 consensus_queue.cc:237] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
W20240501 14:10:34.370712 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: db0f01ee72b24ec9b84410cf94ef9759. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:34.371315 19376 raft_consensus.cc:1261] T db0f01ee72b24ec9b84410cf94ef9759 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:34.371577 19769 raft_consensus.cc:1261] T db0f01ee72b24ec9b84410cf94ef9759 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:34.371721 20654 consensus_queue.cc:1035] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:34.372047 20654 consensus_queue.cc:1035] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 14:10:34.375463 19571 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135:38751): Couldn't send request to peer 43ce4a4c954f4a5f80954b38933c189d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: db0f01ee72b24ec9b84410cf94ef9759. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:34.375347 20654 raft_consensus.cc:2932] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:34.377672 19376 raft_consensus.cc:2932] T db0f01ee72b24ec9b84410cf94ef9759 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:34.377866 19769 raft_consensus.cc:2932] T db0f01ee72b24ec9b84410cf94ef9759 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:34.379925 19232 catalog_manager.cc:5484] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 reported cstate change: config changed from index 4 to 5, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:34.416004 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:34.425295 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:34.431941 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:34.438257 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:34.443724 21248 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:34.447921 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:34.452185 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:34.453959 19376 consensus_queue.cc:237] T 0988c468757d49738765bdcb31794bd1 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
W20240501 14:10:34.455955 19312 consensus_peers.cc:485] T 0988c468757d49738765bdcb31794bd1 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133:46883): Couldn't send request to peer 5bf81f3d986e428190d3f7727f36d96c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0988c468757d49738765bdcb31794bd1. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:34.456593 19769 raft_consensus.cc:1261] T 0988c468757d49738765bdcb31794bd1 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:34.459580 19637 raft_consensus.cc:1261] T 0988c468757d49738765bdcb31794bd1 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:34.460036 22527 consensus_queue.cc:1035] T 0988c468757d49738765bdcb31794bd1 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:34.461310 22527 consensus_queue.cc:1035] T 0988c468757d49738765bdcb31794bd1 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:34.473693 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:34.485705 22527 raft_consensus.cc:2932] T 0988c468757d49738765bdcb31794bd1 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:34.486060 19769 raft_consensus.cc:2932] T 0988c468757d49738765bdcb31794bd1 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:34.488936 19637 raft_consensus.cc:2932] T 0988c468757d49738765bdcb31794bd1 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:34.489212 19233 catalog_manager.cc:5484] T 0988c468757d49738765bdcb31794bd1 P da3c4e948ff747678f18456c134d7ed4 reported cstate change: config changed from index -1 to 4, VOTER 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133) added. New cstate: current_term: 1 leader_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:34.532100 19376 consensus_queue.cc:237] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:34.536216 19769 raft_consensus.cc:1261] T 814ba2247a314a4797aa0f9f624aaeb6 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:34.536777 19506 raft_consensus.cc:1261] T 814ba2247a314a4797aa0f9f624aaeb6 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:10:34.538785 19310 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 814ba2247a314a4797aa0f9f624aaeb6. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:34.541741 22524 consensus_queue.cc:1035] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:34.542660 22524 consensus_queue.cc:1035] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:34.545841 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:34.550195 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:34.563269 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:34.563323 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:34.570799 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:34.580332 21248 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:34.583535 19506 consensus_queue.cc:237] T a9edf1764b55420ea5d519e07ae1d5e0 P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:10:34.586134 19637 raft_consensus.cc:1261] T a9edf1764b55420ea5d519e07ae1d5e0 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Refusing update from remote peer 99a0be6ef2144b7fb247ba9d3f051d66: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:34.587487 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:34.587560 19768 raft_consensus.cc:1261] T a9edf1764b55420ea5d519e07ae1d5e0 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Refusing update from remote peer 99a0be6ef2144b7fb247ba9d3f051d66: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:34.588061 22362 consensus_queue.cc:1035] T a9edf1764b55420ea5d519e07ae1d5e0 P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:34.588418 22081 consensus_queue.cc:1035] T a9edf1764b55420ea5d519e07ae1d5e0 P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:10:34.591723 19440 consensus_peers.cc:485] T a9edf1764b55420ea5d519e07ae1d5e0 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136:34895): Couldn't send request to peer 778c78105092428bbcde75ab2cdb6a05. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a9edf1764b55420ea5d519e07ae1d5e0. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:34.597132 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:34.614090 22081 raft_consensus.cc:2932] T a9edf1764b55420ea5d519e07ae1d5e0 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:34.616751 19637 raft_consensus.cc:2932] T a9edf1764b55420ea5d519e07ae1d5e0 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:34.619305 19226 catalog_manager.cc:5484] T a9edf1764b55420ea5d519e07ae1d5e0 P ec8cde17a00645e7b38d5d29f7425367 reported cstate change: config changed from index -1 to 4, VOTER 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136) added. New cstate: current_term: 1 leader_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:34.621163 19767 raft_consensus.cc:2932] T a9edf1764b55420ea5d519e07ae1d5e0 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:34.624766 22524 raft_consensus.cc:2932] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:34.625352 19769 raft_consensus.cc:2932] T 814ba2247a314a4797aa0f9f624aaeb6 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:34.627291 19506 raft_consensus.cc:2932] T 814ba2247a314a4797aa0f9f624aaeb6 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:34.627959 19234 catalog_manager.cc:5484] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc reported cstate change: config changed from index -1 to 4, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New cstate: current_term: 1 leader_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:34.651604 22733 ts_tablet_manager.cc:902] T cf62635e9a994259b9e293e5eb425b55 P 778c78105092428bbcde75ab2cdb6a05: Initiating tablet copy from peer da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357)
I20240501 14:10:34.652510 22733 tablet_copy_client.cc:308] T cf62635e9a994259b9e293e5eb425b55 P 778c78105092428bbcde75ab2cdb6a05: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.132:35357
I20240501 14:10:34.658726 19789 tablet_copy_service.cc:140] P da3c4e948ff747678f18456c134d7ed4: Received BeginTabletCopySession request for tablet cf62635e9a994259b9e293e5eb425b55 from peer 778c78105092428bbcde75ab2cdb6a05 ({username='slave'} at 127.7.39.136:40943)
I20240501 14:10:34.658908 19789 tablet_copy_service.cc:161] P da3c4e948ff747678f18456c134d7ed4: Beginning new tablet copy session on tablet cf62635e9a994259b9e293e5eb425b55 from peer 778c78105092428bbcde75ab2cdb6a05 at {username='slave'} at 127.7.39.136:40943: session id = 778c78105092428bbcde75ab2cdb6a05-cf62635e9a994259b9e293e5eb425b55
I20240501 14:10:34.660197 19789 tablet_copy_source_session.cc:205] T cf62635e9a994259b9e293e5eb425b55 P da3c4e948ff747678f18456c134d7ed4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:34.664208 22733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf62635e9a994259b9e293e5eb425b55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:34.670114 19506 consensus_queue.cc:237] T 55830424a98348e382759c54afdeabcd P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:10:34.671672 19637 raft_consensus.cc:1261] T 55830424a98348e382759c54afdeabcd P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Refusing update from remote peer 99a0be6ef2144b7fb247ba9d3f051d66: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:34.672621 22712 consensus_queue.cc:1035] T 55830424a98348e382759c54afdeabcd P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:10:34.673185 19440 consensus_peers.cc:485] T 55830424a98348e382759c54afdeabcd P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136:34895): Couldn't send request to peer 778c78105092428bbcde75ab2cdb6a05. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 55830424a98348e382759c54afdeabcd. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:34.674016 19376 raft_consensus.cc:1261] T 55830424a98348e382759c54afdeabcd P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Refusing update from remote peer 99a0be6ef2144b7fb247ba9d3f051d66: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:34.674674 22712 consensus_queue.cc:1035] T 55830424a98348e382759c54afdeabcd P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:34.677824 22733 tablet_copy_client.cc:783] T cf62635e9a994259b9e293e5eb425b55 P 778c78105092428bbcde75ab2cdb6a05: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:34.678200 22733 tablet_copy_client.cc:647] T cf62635e9a994259b9e293e5eb425b55 P 778c78105092428bbcde75ab2cdb6a05: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:34.680358 22733 tablet_copy_client.cc:523] T cf62635e9a994259b9e293e5eb425b55 P 778c78105092428bbcde75ab2cdb6a05: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:34.682471 22733 tablet_bootstrap.cc:492] T cf62635e9a994259b9e293e5eb425b55 P 778c78105092428bbcde75ab2cdb6a05: Bootstrap starting.
I20240501 14:10:34.689484 22712 raft_consensus.cc:2932] T 55830424a98348e382759c54afdeabcd P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:34.690707 19637 raft_consensus.cc:2932] T 55830424a98348e382759c54afdeabcd P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:34.692811 19226 catalog_manager.cc:5484] T 55830424a98348e382759c54afdeabcd P 99a0be6ef2144b7fb247ba9d3f051d66 reported cstate change: config changed from index -1 to 4, VOTER 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136) added. New cstate: current_term: 1 leader_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:34.694116 19376 raft_consensus.cc:2932] T 55830424a98348e382759c54afdeabcd P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:34.695217 22778 ts_tablet_manager.cc:902] T b6be9a115fe14c098c29cad819849f95 P 43ce4a4c954f4a5f80954b38933c189d: Initiating tablet copy from peer 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829)
I20240501 14:10:34.696645 22778 tablet_copy_client.cc:308] T b6be9a115fe14c098c29cad819849f95 P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.130:41829
I20240501 14:10:34.697832 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:34.702476 19769 consensus_queue.cc:237] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:34.703559 19527 tablet_copy_service.cc:140] P 99a0be6ef2144b7fb247ba9d3f051d66: Received BeginTabletCopySession request for tablet b6be9a115fe14c098c29cad819849f95 from peer 43ce4a4c954f4a5f80954b38933c189d ({username='slave'} at 127.7.39.135:44769)
I20240501 14:10:34.703716 19527 tablet_copy_service.cc:161] P 99a0be6ef2144b7fb247ba9d3f051d66: Beginning new tablet copy session on tablet b6be9a115fe14c098c29cad819849f95 from peer 43ce4a4c954f4a5f80954b38933c189d at {username='slave'} at 127.7.39.135:44769: session id = 43ce4a4c954f4a5f80954b38933c189d-b6be9a115fe14c098c29cad819849f95
I20240501 14:10:34.704814 19376 raft_consensus.cc:1261] T 11f64b59f83e458f8cf51fd5bcd36fee P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:34.705263 19527 tablet_copy_source_session.cc:205] T b6be9a115fe14c098c29cad819849f95 P 99a0be6ef2144b7fb247ba9d3f051d66: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:34.707207 22417 consensus_queue.cc:1035] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:34.709415 19638 consensus_queue.cc:237] T 10aa2feeb0324aa4a6221d2f2f3fc941 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
W20240501 14:10:34.710368 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 11f64b59f83e458f8cf51fd5bcd36fee. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:34.710965 19506 raft_consensus.cc:1261] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:34.711464 22574 consensus_queue.cc:1035] T 10aa2feeb0324aa4a6221d2f2f3fc941 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:34.711655 19375 raft_consensus.cc:1261] T 10aa2feeb0324aa4a6221d2f2f3fc941 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:34.712328 22574 consensus_queue.cc:1035] T 10aa2feeb0324aa4a6221d2f2f3fc941 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:34.712404 22778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6be9a115fe14c098c29cad819849f95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:34.715145 19637 raft_consensus.cc:1261] T 11f64b59f83e458f8cf51fd5bcd36fee P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:34.715610 22417 consensus_queue.cc:1035] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:10:34.720453 19571 consensus_peers.cc:485] T 10aa2feeb0324aa4a6221d2f2f3fc941 P ec8cde17a00645e7b38d5d29f7425367 -> Peer 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136:34895): Couldn't send request to peer 778c78105092428bbcde75ab2cdb6a05. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 10aa2feeb0324aa4a6221d2f2f3fc941. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:34.721710 22574 raft_consensus.cc:2932] T 10aa2feeb0324aa4a6221d2f2f3fc941 P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:34.722674 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:34.724067 19506 raft_consensus.cc:2932] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:34.726033 19226 catalog_manager.cc:5484] T 10aa2feeb0324aa4a6221d2f2f3fc941 P ec8cde17a00645e7b38d5d29f7425367 reported cstate change: config changed from index -1 to 4, VOTER 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136) added. New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:34.727103 19375 raft_consensus.cc:2932] T 10aa2feeb0324aa4a6221d2f2f3fc941 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:34.729882 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:34.747959 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:34.752243 22733 log.cc:824] T cf62635e9a994259b9e293e5eb425b55 P 778c78105092428bbcde75ab2cdb6a05: Log is configured to *not* fsync() on all Append() calls
I20240501 14:10:34.771168 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:34.777587 21248 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:34.783130 22778 tablet_copy_client.cc:783] T b6be9a115fe14c098c29cad819849f95 P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:34.783495 22778 tablet_copy_client.cc:647] T b6be9a115fe14c098c29cad819849f95 P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:34.785495 22778 tablet_copy_client.cc:523] T b6be9a115fe14c098c29cad819849f95 P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:34.787676 22778 tablet_bootstrap.cc:492] T b6be9a115fe14c098c29cad819849f95 P 43ce4a4c954f4a5f80954b38933c189d: Bootstrap starting.
I20240501 14:10:34.794888 22778 log.cc:824] T b6be9a115fe14c098c29cad819849f95 P 43ce4a4c954f4a5f80954b38933c189d: Log is configured to *not* fsync() on all Append() calls
I20240501 14:10:34.800930 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:34.801709 22800 ts_tablet_manager.cc:902] T f69020bb6c244848a6b0a50fd2b4f3eb P 5bf81f3d986e428190d3f7727f36d96c: Initiating tablet copy from peer fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847)
I20240501 14:10:34.804239 22800 tablet_copy_client.cc:308] T f69020bb6c244848a6b0a50fd2b4f3eb P 5bf81f3d986e428190d3f7727f36d96c: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.129:35847
I20240501 14:10:34.806465 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:34.806874 22817 ts_tablet_manager.cc:902] T ad7c255baee74e629e2976a942a4174f P 43ce4a4c954f4a5f80954b38933c189d: Initiating tablet copy from peer fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847)
I20240501 14:10:34.807448 22817 tablet_copy_client.cc:308] T ad7c255baee74e629e2976a942a4174f P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.129:35847
I20240501 14:10:34.814205 19396 tablet_copy_service.cc:140] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Received BeginTabletCopySession request for tablet ad7c255baee74e629e2976a942a4174f from peer 43ce4a4c954f4a5f80954b38933c189d ({username='slave'} at 127.7.39.135:49533)
I20240501 14:10:34.814349 19396 tablet_copy_service.cc:161] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Beginning new tablet copy session on tablet ad7c255baee74e629e2976a942a4174f from peer 43ce4a4c954f4a5f80954b38933c189d at {username='slave'} at 127.7.39.135:49533: session id = 43ce4a4c954f4a5f80954b38933c189d-ad7c255baee74e629e2976a942a4174f
I20240501 14:10:34.815959 19396 tablet_copy_source_session.cc:205] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:34.819950 22817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad7c255baee74e629e2976a942a4174f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:34.840821 22838 ts_tablet_manager.cc:902] T db0f01ee72b24ec9b84410cf94ef9759 P 43ce4a4c954f4a5f80954b38933c189d: Initiating tablet copy from peer ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201)
I20240501 14:10:34.842759 22838 tablet_copy_client.cc:308] T db0f01ee72b24ec9b84410cf94ef9759 P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.131:45201
I20240501 14:10:34.848486 19396 tablet_copy_service.cc:140] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Received BeginTabletCopySession request for tablet f69020bb6c244848a6b0a50fd2b4f3eb from peer 5bf81f3d986e428190d3f7727f36d96c ({username='slave'} at 127.7.39.133:53931)
I20240501 14:10:34.848623 19396 tablet_copy_service.cc:161] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Beginning new tablet copy session on tablet f69020bb6c244848a6b0a50fd2b4f3eb from peer 5bf81f3d986e428190d3f7727f36d96c at {username='slave'} at 127.7.39.133:53931: session id = 5bf81f3d986e428190d3f7727f36d96c-f69020bb6c244848a6b0a50fd2b4f3eb
I20240501 14:10:34.849696 19396 tablet_copy_source_session.cc:205] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:34.852977 19658 tablet_copy_service.cc:140] P ec8cde17a00645e7b38d5d29f7425367: Received BeginTabletCopySession request for tablet db0f01ee72b24ec9b84410cf94ef9759 from peer 43ce4a4c954f4a5f80954b38933c189d ({username='slave'} at 127.7.39.135:35759)
I20240501 14:10:34.853137 19658 tablet_copy_service.cc:161] P ec8cde17a00645e7b38d5d29f7425367: Beginning new tablet copy session on tablet db0f01ee72b24ec9b84410cf94ef9759 from peer 43ce4a4c954f4a5f80954b38933c189d at {username='slave'} at 127.7.39.135:35759: session id = 43ce4a4c954f4a5f80954b38933c189d-db0f01ee72b24ec9b84410cf94ef9759
I20240501 14:10:34.853780 22733 tablet_bootstrap.cc:492] T cf62635e9a994259b9e293e5eb425b55 P 778c78105092428bbcde75ab2cdb6a05: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:34.854180 22733 tablet_bootstrap.cc:492] T cf62635e9a994259b9e293e5eb425b55 P 778c78105092428bbcde75ab2cdb6a05: Bootstrap complete.
I20240501 14:10:34.854563 22733 ts_tablet_manager.cc:1367] T cf62635e9a994259b9e293e5eb425b55 P 778c78105092428bbcde75ab2cdb6a05: Time spent bootstrapping tablet: real 0.172s user 0.046s sys 0.000s
I20240501 14:10:34.854777 19658 tablet_copy_source_session.cc:205] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:34.857563 22733 raft_consensus.cc:348] T cf62635e9a994259b9e293e5eb425b55 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:10:34.857825 22817 tablet_copy_client.cc:783] T ad7c255baee74e629e2976a942a4174f P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:34.857834 22733 raft_consensus.cc:729] T cf62635e9a994259b9e293e5eb425b55 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 778c78105092428bbcde75ab2cdb6a05, State: Initialized, Role: FOLLOWER
I20240501 14:10:34.858104 22800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f69020bb6c244848a6b0a50fd2b4f3eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:34.858379 22733 consensus_queue.cc:260] T cf62635e9a994259b9e293e5eb425b55 P 778c78105092428bbcde75ab2cdb6a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:10:34.858834 22817 tablet_copy_client.cc:647] T ad7c255baee74e629e2976a942a4174f P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:34.859323 22733 ts_tablet_manager.cc:1398] T cf62635e9a994259b9e293e5eb425b55 P 778c78105092428bbcde75ab2cdb6a05: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 14:10:34.859903 22865 ts_tablet_manager.cc:902] T 7eea58d4a33f422a9ada1e82d408577f P 43ce4a4c954f4a5f80954b38933c189d: Initiating tablet copy from peer da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357)
I20240501 14:10:34.860126 19789 tablet_copy_service.cc:342] P da3c4e948ff747678f18456c134d7ed4: Request end of tablet copy session 778c78105092428bbcde75ab2cdb6a05-cf62635e9a994259b9e293e5eb425b55 received from {username='slave'} at 127.7.39.136:40943
I20240501 14:10:34.860226 19789 tablet_copy_service.cc:433] P da3c4e948ff747678f18456c134d7ed4: ending tablet copy session 778c78105092428bbcde75ab2cdb6a05-cf62635e9a994259b9e293e5eb425b55 on tablet cf62635e9a994259b9e293e5eb425b55 with peer 778c78105092428bbcde75ab2cdb6a05
I20240501 14:10:34.861670 22865 tablet_copy_client.cc:308] T 7eea58d4a33f422a9ada1e82d408577f P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.132:35357
I20240501 14:10:34.864127 22838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db0f01ee72b24ec9b84410cf94ef9759. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:34.866464 19789 tablet_copy_service.cc:140] P da3c4e948ff747678f18456c134d7ed4: Received BeginTabletCopySession request for tablet 7eea58d4a33f422a9ada1e82d408577f from peer 43ce4a4c954f4a5f80954b38933c189d ({username='slave'} at 127.7.39.135:46639)
I20240501 14:10:34.866585 19789 tablet_copy_service.cc:161] P da3c4e948ff747678f18456c134d7ed4: Beginning new tablet copy session on tablet 7eea58d4a33f422a9ada1e82d408577f from peer 43ce4a4c954f4a5f80954b38933c189d at {username='slave'} at 127.7.39.135:46639: session id = 43ce4a4c954f4a5f80954b38933c189d-7eea58d4a33f422a9ada1e82d408577f
I20240501 14:10:34.866820 22817 tablet_copy_client.cc:523] T ad7c255baee74e629e2976a942a4174f P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:34.867679 19789 tablet_copy_source_session.cc:205] T 7eea58d4a33f422a9ada1e82d408577f P da3c4e948ff747678f18456c134d7ed4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:34.868266 22817 tablet_bootstrap.cc:492] T ad7c255baee74e629e2976a942a4174f P 43ce4a4c954f4a5f80954b38933c189d: Bootstrap starting.
I20240501 14:10:34.869441 22417 raft_consensus.cc:2932] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:34.870628 22865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7eea58d4a33f422a9ada1e82d408577f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:34.872406 19637 raft_consensus.cc:2932] T 11f64b59f83e458f8cf51fd5bcd36fee P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:34.873258 19226 catalog_manager.cc:5484] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 reported cstate change: config changed from index -1 to 4, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New cstate: current_term: 1 leader_uuid: "da3c4e948ff747678f18456c134d7ed4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:34.874485 19376 raft_consensus.cc:2932] T 11f64b59f83e458f8cf51fd5bcd36fee P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:34.878576 22883 ts_tablet_manager.cc:902] T 0988c468757d49738765bdcb31794bd1 P 5bf81f3d986e428190d3f7727f36d96c: Initiating tablet copy from peer fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847)
I20240501 14:10:34.879186 22883 tablet_copy_client.cc:308] T 0988c468757d49738765bdcb31794bd1 P 5bf81f3d986e428190d3f7727f36d96c: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.129:35847
I20240501 14:10:34.879771 19396 tablet_copy_service.cc:140] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Received BeginTabletCopySession request for tablet 0988c468757d49738765bdcb31794bd1 from peer 5bf81f3d986e428190d3f7727f36d96c ({username='slave'} at 127.7.39.133:53931)
I20240501 14:10:34.879878 19396 tablet_copy_service.cc:161] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Beginning new tablet copy session on tablet 0988c468757d49738765bdcb31794bd1 from peer 5bf81f3d986e428190d3f7727f36d96c at {username='slave'} at 127.7.39.133:53931: session id = 5bf81f3d986e428190d3f7727f36d96c-0988c468757d49738765bdcb31794bd1
I20240501 14:10:34.880784 19396 tablet_copy_source_session.cc:205] T 0988c468757d49738765bdcb31794bd1 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:34.882892 22883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0988c468757d49738765bdcb31794bd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:34.887542 22838 tablet_copy_client.cc:783] T db0f01ee72b24ec9b84410cf94ef9759 P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:34.887825 22838 tablet_copy_client.cc:647] T db0f01ee72b24ec9b84410cf94ef9759 P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:34.888094 22865 tablet_copy_client.cc:783] T 7eea58d4a33f422a9ada1e82d408577f P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:34.888262 22865 tablet_copy_client.cc:647] T 7eea58d4a33f422a9ada1e82d408577f P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:34.888564 22871 ts_tablet_manager.cc:902] T 85593473500b4d5eaf9f8b0bf2296777 P e6246b183293412494919f1e7efc9448: Initiating tablet copy from peer ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201)
I20240501 14:10:34.889566 22871 tablet_copy_client.cc:308] T 85593473500b4d5eaf9f8b0bf2296777 P e6246b183293412494919f1e7efc9448: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.131:45201
I20240501 14:10:34.889765 22883 tablet_copy_client.cc:783] T 0988c468757d49738765bdcb31794bd1 P 5bf81f3d986e428190d3f7727f36d96c: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:34.889983 22883 tablet_copy_client.cc:647] T 0988c468757d49738765bdcb31794bd1 P 5bf81f3d986e428190d3f7727f36d96c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:34.891595 22838 tablet_copy_client.cc:523] T db0f01ee72b24ec9b84410cf94ef9759 P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:34.892678 22778 tablet_bootstrap.cc:492] T b6be9a115fe14c098c29cad819849f95 P 43ce4a4c954f4a5f80954b38933c189d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:34.892997 22778 tablet_bootstrap.cc:492] T b6be9a115fe14c098c29cad819849f95 P 43ce4a4c954f4a5f80954b38933c189d: Bootstrap complete.
I20240501 14:10:34.893182 22838 tablet_bootstrap.cc:492] T db0f01ee72b24ec9b84410cf94ef9759 P 43ce4a4c954f4a5f80954b38933c189d: Bootstrap starting.
I20240501 14:10:34.893321 22778 ts_tablet_manager.cc:1367] T b6be9a115fe14c098c29cad819849f95 P 43ce4a4c954f4a5f80954b38933c189d: Time spent bootstrapping tablet: real 0.106s user 0.030s sys 0.004s
I20240501 14:10:34.895829 22778 raft_consensus.cc:348] T b6be9a115fe14c098c29cad819849f95 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:34.896035 22778 raft_consensus.cc:729] T b6be9a115fe14c098c29cad819849f95 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43ce4a4c954f4a5f80954b38933c189d, State: Initialized, Role: FOLLOWER
I20240501 14:10:34.896463 22778 consensus_queue.cc:260] T b6be9a115fe14c098c29cad819849f95 P 43ce4a4c954f4a5f80954b38933c189d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:34.898334 22778 ts_tablet_manager.cc:1398] T b6be9a115fe14c098c29cad819849f95 P 43ce4a4c954f4a5f80954b38933c189d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240501 14:10:34.899078 19527 tablet_copy_service.cc:342] P 99a0be6ef2144b7fb247ba9d3f051d66: Request end of tablet copy session 43ce4a4c954f4a5f80954b38933c189d-b6be9a115fe14c098c29cad819849f95 received from {username='slave'} at 127.7.39.135:44769
I20240501 14:10:34.899173 19527 tablet_copy_service.cc:433] P 99a0be6ef2144b7fb247ba9d3f051d66: ending tablet copy session 43ce4a4c954f4a5f80954b38933c189d-b6be9a115fe14c098c29cad819849f95 on tablet b6be9a115fe14c098c29cad819849f95 with peer 43ce4a4c954f4a5f80954b38933c189d
I20240501 14:10:34.902298 22865 tablet_copy_client.cc:523] T 7eea58d4a33f422a9ada1e82d408577f P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:34.902891 22883 tablet_copy_client.cc:523] T 0988c468757d49738765bdcb31794bd1 P 5bf81f3d986e428190d3f7727f36d96c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:34.903854 22865 tablet_bootstrap.cc:492] T 7eea58d4a33f422a9ada1e82d408577f P 43ce4a4c954f4a5f80954b38933c189d: Bootstrap starting.
I20240501 14:10:34.905903 19658 tablet_copy_service.cc:140] P ec8cde17a00645e7b38d5d29f7425367: Received BeginTabletCopySession request for tablet 85593473500b4d5eaf9f8b0bf2296777 from peer e6246b183293412494919f1e7efc9448 ({username='slave'} at 127.7.39.134:41913)
I20240501 14:10:34.906028 19658 tablet_copy_service.cc:161] P ec8cde17a00645e7b38d5d29f7425367: Beginning new tablet copy session on tablet 85593473500b4d5eaf9f8b0bf2296777 from peer e6246b183293412494919f1e7efc9448 at {username='slave'} at 127.7.39.134:41913: session id = e6246b183293412494919f1e7efc9448-85593473500b4d5eaf9f8b0bf2296777
I20240501 14:10:34.906912 22883 tablet_bootstrap.cc:492] T 0988c468757d49738765bdcb31794bd1 P 5bf81f3d986e428190d3f7727f36d96c: Bootstrap starting.
I20240501 14:10:34.907013 19658 tablet_copy_source_session.cc:205] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:34.909379 22800 tablet_copy_client.cc:783] T f69020bb6c244848a6b0a50fd2b4f3eb P 5bf81f3d986e428190d3f7727f36d96c: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:34.909593 22800 tablet_copy_client.cc:647] T f69020bb6c244848a6b0a50fd2b4f3eb P 5bf81f3d986e428190d3f7727f36d96c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:34.912434 22778 ts_tablet_manager.cc:902] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 43ce4a4c954f4a5f80954b38933c189d: Initiating tablet copy from peer fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847)
I20240501 14:10:34.912902 22778 tablet_copy_client.cc:308] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.129:35847
I20240501 14:10:34.913358 19396 tablet_copy_service.cc:140] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Received BeginTabletCopySession request for tablet 98af8b06b9324e3fa7cdb898b7b9fab9 from peer 43ce4a4c954f4a5f80954b38933c189d ({username='slave'} at 127.7.39.135:49533)
I20240501 14:10:34.913460 19396 tablet_copy_service.cc:161] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Beginning new tablet copy session on tablet 98af8b06b9324e3fa7cdb898b7b9fab9 from peer 43ce4a4c954f4a5f80954b38933c189d at {username='slave'} at 127.7.39.135:49533: session id = 43ce4a4c954f4a5f80954b38933c189d-98af8b06b9324e3fa7cdb898b7b9fab9
I20240501 14:10:34.914391 19396 tablet_copy_source_session.cc:205] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:34.915618 22778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98af8b06b9324e3fa7cdb898b7b9fab9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:34.918135 22778 tablet_copy_client.cc:783] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:34.918292 22778 tablet_copy_client.cc:647] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:34.918314 22871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85593473500b4d5eaf9f8b0bf2296777. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:34.924245 22800 tablet_copy_client.cc:523] T f69020bb6c244848a6b0a50fd2b4f3eb P 5bf81f3d986e428190d3f7727f36d96c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:34.925940 22800 tablet_bootstrap.cc:492] T f69020bb6c244848a6b0a50fd2b4f3eb P 5bf81f3d986e428190d3f7727f36d96c: Bootstrap starting.
I20240501 14:10:34.933168 22883 log.cc:824] T 0988c468757d49738765bdcb31794bd1 P 5bf81f3d986e428190d3f7727f36d96c: Log is configured to *not* fsync() on all Append() calls
I20240501 14:10:34.934419 22912 ts_tablet_manager.cc:902] T 7eea58d4a33f422a9ada1e82d408577f P 5bf81f3d986e428190d3f7727f36d96c: Initiating tablet copy from peer da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357)
I20240501 14:10:34.935214 22912 tablet_copy_client.cc:308] T 7eea58d4a33f422a9ada1e82d408577f P 5bf81f3d986e428190d3f7727f36d96c: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.132:35357
I20240501 14:10:34.938855 22817 tablet_bootstrap.cc:492] T ad7c255baee74e629e2976a942a4174f P 43ce4a4c954f4a5f80954b38933c189d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:34.939183 22817 tablet_bootstrap.cc:492] T ad7c255baee74e629e2976a942a4174f P 43ce4a4c954f4a5f80954b38933c189d: Bootstrap complete.
I20240501 14:10:34.941490 19789 tablet_copy_service.cc:140] P da3c4e948ff747678f18456c134d7ed4: Received BeginTabletCopySession request for tablet 7eea58d4a33f422a9ada1e82d408577f from peer 5bf81f3d986e428190d3f7727f36d96c ({username='slave'} at 127.7.39.133:51567)
I20240501 14:10:34.941627 19789 tablet_copy_service.cc:161] P da3c4e948ff747678f18456c134d7ed4: Beginning new tablet copy session on tablet 7eea58d4a33f422a9ada1e82d408577f from peer 5bf81f3d986e428190d3f7727f36d96c at {username='slave'} at 127.7.39.133:51567: session id = 5bf81f3d986e428190d3f7727f36d96c-7eea58d4a33f422a9ada1e82d408577f
I20240501 14:10:34.942672 19789 tablet_copy_source_session.cc:205] T 7eea58d4a33f422a9ada1e82d408577f P da3c4e948ff747678f18456c134d7ed4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:34.940954 22778 tablet_copy_client.cc:523] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:34.946262 22912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7eea58d4a33f422a9ada1e82d408577f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:34.947273 22778 tablet_bootstrap.cc:492] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 43ce4a4c954f4a5f80954b38933c189d: Bootstrap starting.
I20240501 14:10:34.949415 22912 tablet_copy_client.cc:783] T 7eea58d4a33f422a9ada1e82d408577f P 5bf81f3d986e428190d3f7727f36d96c: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:34.949640 22912 tablet_copy_client.cc:647] T 7eea58d4a33f422a9ada1e82d408577f P 5bf81f3d986e428190d3f7727f36d96c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:34.948967 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:34.939355 22817 ts_tablet_manager.cc:1367] T ad7c255baee74e629e2976a942a4174f P 43ce4a4c954f4a5f80954b38933c189d: Time spent bootstrapping tablet: real 0.071s user 0.029s sys 0.001s
I20240501 14:10:34.951330 22817 raft_consensus.cc:348] T ad7c255baee74e629e2976a942a4174f P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:34.951503 22817 raft_consensus.cc:729] T ad7c255baee74e629e2976a942a4174f P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43ce4a4c954f4a5f80954b38933c189d, State: Initialized, Role: FOLLOWER
I20240501 14:10:34.951690 22817 consensus_queue.cc:260] T ad7c255baee74e629e2976a942a4174f P 43ce4a4c954f4a5f80954b38933c189d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:34.952025 22817 ts_tablet_manager.cc:1398] T ad7c255baee74e629e2976a942a4174f P 43ce4a4c954f4a5f80954b38933c189d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:10:34.953910 19396 tablet_copy_service.cc:342] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Request end of tablet copy session 43ce4a4c954f4a5f80954b38933c189d-ad7c255baee74e629e2976a942a4174f received from {username='slave'} at 127.7.39.135:49533
I20240501 14:10:34.954028 19396 tablet_copy_service.cc:433] P fb1e6bc2d07b4ce3b622a66f14c93bdc: ending tablet copy session 43ce4a4c954f4a5f80954b38933c189d-ad7c255baee74e629e2976a942a4174f on tablet ad7c255baee74e629e2976a942a4174f with peer 43ce4a4c954f4a5f80954b38933c189d
I20240501 14:10:34.957015 22912 tablet_copy_client.cc:523] T 7eea58d4a33f422a9ada1e82d408577f P 5bf81f3d986e428190d3f7727f36d96c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:34.958451 22912 tablet_bootstrap.cc:492] T 7eea58d4a33f422a9ada1e82d408577f P 5bf81f3d986e428190d3f7727f36d96c: Bootstrap starting.
I20240501 14:10:34.967868 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:34.992921 22778 tablet_bootstrap.cc:492] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 43ce4a4c954f4a5f80954b38933c189d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:34.993225 22778 tablet_bootstrap.cc:492] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 43ce4a4c954f4a5f80954b38933c189d: Bootstrap complete.
I20240501 14:10:34.993530 22778 ts_tablet_manager.cc:1367] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 43ce4a4c954f4a5f80954b38933c189d: Time spent bootstrapping tablet: real 0.046s user 0.022s sys 0.004s
I20240501 14:10:34.994117 22778 raft_consensus.cc:348] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:34.994261 22778 raft_consensus.cc:729] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43ce4a4c954f4a5f80954b38933c189d, State: Initialized, Role: FOLLOWER
I20240501 14:10:34.994426 22778 consensus_queue.cc:260] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 43ce4a4c954f4a5f80954b38933c189d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:34.994699 22778 ts_tablet_manager.cc:1398] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 43ce4a4c954f4a5f80954b38933c189d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:34.996443 19396 tablet_copy_service.cc:342] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Request end of tablet copy session 43ce4a4c954f4a5f80954b38933c189d-98af8b06b9324e3fa7cdb898b7b9fab9 received from {username='slave'} at 127.7.39.135:49533
I20240501 14:10:34.996532 19396 tablet_copy_service.cc:433] P fb1e6bc2d07b4ce3b622a66f14c93bdc: ending tablet copy session 43ce4a4c954f4a5f80954b38933c189d-98af8b06b9324e3fa7cdb898b7b9fab9 on tablet 98af8b06b9324e3fa7cdb898b7b9fab9 with peer 43ce4a4c954f4a5f80954b38933c189d
I20240501 14:10:34.998772 21248 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:34.999714 19637 consensus_queue.cc:237] T 10aa2feeb0324aa4a6221d2f2f3fc941 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:35.001513 19376 raft_consensus.cc:1261] T 10aa2feeb0324aa4a6221d2f2f3fc941 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:35.001973 19506 raft_consensus.cc:1261] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 14:10:35.002624 19571 consensus_peers.cc:485] T 10aa2feeb0324aa4a6221d2f2f3fc941 P ec8cde17a00645e7b38d5d29f7425367 -> Peer 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136:34895): Couldn't send request to peer 778c78105092428bbcde75ab2cdb6a05. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 10aa2feeb0324aa4a6221d2f2f3fc941. This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:35.003180 19571 consensus_peers.cc:485] T 10aa2feeb0324aa4a6221d2f2f3fc941 P ec8cde17a00645e7b38d5d29f7425367 -> Peer 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135:38751): Couldn't send request to peer 43ce4a4c954f4a5f80954b38933c189d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 10aa2feeb0324aa4a6221d2f2f3fc941. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:35.003784 22086 consensus_queue.cc:1035] T 10aa2feeb0324aa4a6221d2f2f3fc941 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:35.003916 20654 consensus_queue.cc:1035] T 10aa2feeb0324aa4a6221d2f2f3fc941 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:35.004076 22865 tablet_bootstrap.cc:492] T 7eea58d4a33f422a9ada1e82d408577f P 43ce4a4c954f4a5f80954b38933c189d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:35.004427 22865 tablet_bootstrap.cc:492] T 7eea58d4a33f422a9ada1e82d408577f P 43ce4a4c954f4a5f80954b38933c189d: Bootstrap complete.
I20240501 14:10:35.004619 22865 ts_tablet_manager.cc:1367] T 7eea58d4a33f422a9ada1e82d408577f P 43ce4a4c954f4a5f80954b38933c189d: Time spent bootstrapping tablet: real 0.101s user 0.021s sys 0.004s
I20240501 14:10:35.005645 22865 raft_consensus.cc:348] T 7eea58d4a33f422a9ada1e82d408577f P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:35.005770 22865 raft_consensus.cc:729] T 7eea58d4a33f422a9ada1e82d408577f P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43ce4a4c954f4a5f80954b38933c189d, State: Initialized, Role: FOLLOWER
I20240501 14:10:35.005934 22865 consensus_queue.cc:260] T 7eea58d4a33f422a9ada1e82d408577f P 43ce4a4c954f4a5f80954b38933c189d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:35.006247 22865 ts_tablet_manager.cc:1398] T 7eea58d4a33f422a9ada1e82d408577f P 43ce4a4c954f4a5f80954b38933c189d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:35.006458 22086 raft_consensus.cc:2932] T 10aa2feeb0324aa4a6221d2f2f3fc941 P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:35.008402 19506 raft_consensus.cc:2932] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:35.012229 19232 catalog_manager.cc:5484] T 10aa2feeb0324aa4a6221d2f2f3fc941 P ec8cde17a00645e7b38d5d29f7425367 reported cstate change: config changed from index 4 to 5, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:35.013635 19376 raft_consensus.cc:2932] T 10aa2feeb0324aa4a6221d2f2f3fc941 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:35.016414 19789 tablet_copy_service.cc:342] P da3c4e948ff747678f18456c134d7ed4: Request end of tablet copy session 43ce4a4c954f4a5f80954b38933c189d-7eea58d4a33f422a9ada1e82d408577f received from {username='slave'} at 127.7.39.135:46639
I20240501 14:10:35.016513 19789 tablet_copy_service.cc:433] P da3c4e948ff747678f18456c134d7ed4: ending tablet copy session 43ce4a4c954f4a5f80954b38933c189d-7eea58d4a33f422a9ada1e82d408577f on tablet 7eea58d4a33f422a9ada1e82d408577f with peer 43ce4a4c954f4a5f80954b38933c189d
I20240501 14:10:35.018946 22883 tablet_bootstrap.cc:492] T 0988c468757d49738765bdcb31794bd1 P 5bf81f3d986e428190d3f7727f36d96c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:35.019364 22883 tablet_bootstrap.cc:492] T 0988c468757d49738765bdcb31794bd1 P 5bf81f3d986e428190d3f7727f36d96c: Bootstrap complete.
I20240501 14:10:35.019716 22883 ts_tablet_manager.cc:1367] T 0988c468757d49738765bdcb31794bd1 P 5bf81f3d986e428190d3f7727f36d96c: Time spent bootstrapping tablet: real 0.113s user 0.038s sys 0.005s
I20240501 14:10:35.022356 22883 raft_consensus.cc:348] T 0988c468757d49738765bdcb31794bd1 P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:35.022614 22883 raft_consensus.cc:729] T 0988c468757d49738765bdcb31794bd1 P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf81f3d986e428190d3f7727f36d96c, State: Initialized, Role: FOLLOWER
I20240501 14:10:35.023136 22883 consensus_queue.cc:260] T 0988c468757d49738765bdcb31794bd1 P 5bf81f3d986e428190d3f7727f36d96c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:35.024111 22883 ts_tablet_manager.cc:1398] T 0988c468757d49738765bdcb31794bd1 P 5bf81f3d986e428190d3f7727f36d96c: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 14:10:35.024875 19396 tablet_copy_service.cc:342] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Request end of tablet copy session 5bf81f3d986e428190d3f7727f36d96c-0988c468757d49738765bdcb31794bd1 received from {username='slave'} at 127.7.39.133:53931
I20240501 14:10:35.024982 19396 tablet_copy_service.cc:433] P fb1e6bc2d07b4ce3b622a66f14c93bdc: ending tablet copy session 5bf81f3d986e428190d3f7727f36d96c-0988c468757d49738765bdcb31794bd1 on tablet 0988c468757d49738765bdcb31794bd1 with peer 5bf81f3d986e428190d3f7727f36d96c
I20240501 14:10:35.027449 22733 ts_tablet_manager.cc:902] T a9edf1764b55420ea5d519e07ae1d5e0 P 778c78105092428bbcde75ab2cdb6a05: Initiating tablet copy from peer 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829)
I20240501 14:10:35.027951 22733 tablet_copy_client.cc:308] T a9edf1764b55420ea5d519e07ae1d5e0 P 778c78105092428bbcde75ab2cdb6a05: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.130:41829
I20240501 14:10:35.033947 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:35.034587 22912 tablet_bootstrap.cc:492] T 7eea58d4a33f422a9ada1e82d408577f P 5bf81f3d986e428190d3f7727f36d96c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:35.035120 22912 tablet_bootstrap.cc:492] T 7eea58d4a33f422a9ada1e82d408577f P 5bf81f3d986e428190d3f7727f36d96c: Bootstrap complete.
I20240501 14:10:35.035378 22912 ts_tablet_manager.cc:1367] T 7eea58d4a33f422a9ada1e82d408577f P 5bf81f3d986e428190d3f7727f36d96c: Time spent bootstrapping tablet: real 0.077s user 0.032s sys 0.000s
I20240501 14:10:35.036024 22912 raft_consensus.cc:348] T 7eea58d4a33f422a9ada1e82d408577f P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:35.036239 22912 raft_consensus.cc:729] T 7eea58d4a33f422a9ada1e82d408577f P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf81f3d986e428190d3f7727f36d96c, State: Initialized, Role: FOLLOWER
I20240501 14:10:35.036456 22912 consensus_queue.cc:260] T 7eea58d4a33f422a9ada1e82d408577f P 5bf81f3d986e428190d3f7727f36d96c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:35.036551 19527 tablet_copy_service.cc:140] P 99a0be6ef2144b7fb247ba9d3f051d66: Received BeginTabletCopySession request for tablet a9edf1764b55420ea5d519e07ae1d5e0 from peer 778c78105092428bbcde75ab2cdb6a05 ({username='slave'} at 127.7.39.136:39441)
I20240501 14:10:35.036650 19527 tablet_copy_service.cc:161] P 99a0be6ef2144b7fb247ba9d3f051d66: Beginning new tablet copy session on tablet a9edf1764b55420ea5d519e07ae1d5e0 from peer 778c78105092428bbcde75ab2cdb6a05 at {username='slave'} at 127.7.39.136:39441: session id = 778c78105092428bbcde75ab2cdb6a05-a9edf1764b55420ea5d519e07ae1d5e0
I20240501 14:10:35.036861 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:35.036963 22912 ts_tablet_manager.cc:1398] T 7eea58d4a33f422a9ada1e82d408577f P 5bf81f3d986e428190d3f7727f36d96c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:35.037426 19789 tablet_copy_service.cc:342] P da3c4e948ff747678f18456c134d7ed4: Request end of tablet copy session 5bf81f3d986e428190d3f7727f36d96c-7eea58d4a33f422a9ada1e82d408577f received from {username='slave'} at 127.7.39.133:51567
I20240501 14:10:35.037528 19789 tablet_copy_service.cc:433] P da3c4e948ff747678f18456c134d7ed4: ending tablet copy session 5bf81f3d986e428190d3f7727f36d96c-7eea58d4a33f422a9ada1e82d408577f on tablet 7eea58d4a33f422a9ada1e82d408577f with peer 5bf81f3d986e428190d3f7727f36d96c
I20240501 14:10:35.037564 22927 ts_tablet_manager.cc:902] T db0f01ee72b24ec9b84410cf94ef9759 P e6246b183293412494919f1e7efc9448: Initiating tablet copy from peer ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201)
I20240501 14:10:35.037652 19527 tablet_copy_source_session.cc:205] T a9edf1764b55420ea5d519e07ae1d5e0 P 99a0be6ef2144b7fb247ba9d3f051d66: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:35.038134 22927 tablet_copy_client.cc:308] T db0f01ee72b24ec9b84410cf94ef9759 P e6246b183293412494919f1e7efc9448: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.131:45201
I20240501 14:10:35.038753 19658 tablet_copy_service.cc:140] P ec8cde17a00645e7b38d5d29f7425367: Received BeginTabletCopySession request for tablet db0f01ee72b24ec9b84410cf94ef9759 from peer e6246b183293412494919f1e7efc9448 ({username='slave'} at 127.7.39.134:41913)
I20240501 14:10:35.038868 19658 tablet_copy_service.cc:161] P ec8cde17a00645e7b38d5d29f7425367: Beginning new tablet copy session on tablet db0f01ee72b24ec9b84410cf94ef9759 from peer e6246b183293412494919f1e7efc9448 at {username='slave'} at 127.7.39.134:41913: session id = e6246b183293412494919f1e7efc9448-db0f01ee72b24ec9b84410cf94ef9759
I20240501 14:10:35.038975 22733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9edf1764b55420ea5d519e07ae1d5e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:35.039913 19658 tablet_copy_source_session.cc:205] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:35.041231 22733 tablet_copy_client.cc:783] T a9edf1764b55420ea5d519e07ae1d5e0 P 778c78105092428bbcde75ab2cdb6a05: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:35.041373 22733 tablet_copy_client.cc:647] T a9edf1764b55420ea5d519e07ae1d5e0 P 778c78105092428bbcde75ab2cdb6a05: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:35.046994 22733 tablet_copy_client.cc:523] T a9edf1764b55420ea5d519e07ae1d5e0 P 778c78105092428bbcde75ab2cdb6a05: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:35.048269 22733 tablet_bootstrap.cc:492] T a9edf1764b55420ea5d519e07ae1d5e0 P 778c78105092428bbcde75ab2cdb6a05: Bootstrap starting.
I20240501 14:10:35.048601 22871 tablet_copy_client.cc:783] T 85593473500b4d5eaf9f8b0bf2296777 P e6246b183293412494919f1e7efc9448: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:35.048823 22871 tablet_copy_client.cc:647] T 85593473500b4d5eaf9f8b0bf2296777 P e6246b183293412494919f1e7efc9448: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:35.060395 19376 consensus_queue.cc:237] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:35.062649 21399 raft_consensus.cc:1261] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 14:10:35.063760 19312 consensus_peers.cc:485] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133:46883): Couldn't send request to peer 5bf81f3d986e428190d3f7727f36d96c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 98af8b06b9324e3fa7cdb898b7b9fab9. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:35.063877 21048 consensus_queue.cc:1035] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:35.064828 19637 raft_consensus.cc:1261] T 98af8b06b9324e3fa7cdb898b7b9fab9 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:35.064862 19506 raft_consensus.cc:1261] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:35.065259 22527 consensus_queue.cc:1035] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:35.065375 22237 consensus_queue.cc:1035] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:35.067207 22527 raft_consensus.cc:2932] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:35.067541 19637 raft_consensus.cc:2932] T 98af8b06b9324e3fa7cdb898b7b9fab9 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:35.069689 22838 tablet_bootstrap.cc:492] T db0f01ee72b24ec9b84410cf94ef9759 P 43ce4a4c954f4a5f80954b38933c189d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:35.070016 22838 tablet_bootstrap.cc:492] T db0f01ee72b24ec9b84410cf94ef9759 P 43ce4a4c954f4a5f80954b38933c189d: Bootstrap complete.
I20240501 14:10:35.070205 22838 ts_tablet_manager.cc:1367] T db0f01ee72b24ec9b84410cf94ef9759 P 43ce4a4c954f4a5f80954b38933c189d: Time spent bootstrapping tablet: real 0.177s user 0.022s sys 0.005s
I20240501 14:10:35.070688 22838 raft_consensus.cc:348] T db0f01ee72b24ec9b84410cf94ef9759 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:35.070657 19232 catalog_manager.cc:5484] T 98af8b06b9324e3fa7cdb898b7b9fab9 P ec8cde17a00645e7b38d5d29f7425367 reported cstate change: config changed from index 4 to 5, VOTER 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133) added. New cstate: current_term: 1 leader_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:35.070806 22838 raft_consensus.cc:729] T db0f01ee72b24ec9b84410cf94ef9759 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43ce4a4c954f4a5f80954b38933c189d, State: Initialized, Role: FOLLOWER
I20240501 14:10:35.070957 22838 consensus_queue.cc:260] T db0f01ee72b24ec9b84410cf94ef9759 P 43ce4a4c954f4a5f80954b38933c189d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:35.071221 22838 ts_tablet_manager.cc:1398] T db0f01ee72b24ec9b84410cf94ef9759 P 43ce4a4c954f4a5f80954b38933c189d: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 14:10:35.073441 19658 tablet_copy_service.cc:342] P ec8cde17a00645e7b38d5d29f7425367: Request end of tablet copy session 43ce4a4c954f4a5f80954b38933c189d-db0f01ee72b24ec9b84410cf94ef9759 received from {username='slave'} at 127.7.39.135:35759
I20240501 14:10:35.073359 19506 raft_consensus.cc:2932] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:35.073532 19658 tablet_copy_service.cc:433] P ec8cde17a00645e7b38d5d29f7425367: ending tablet copy session 43ce4a4c954f4a5f80954b38933c189d-db0f01ee72b24ec9b84410cf94ef9759 on tablet db0f01ee72b24ec9b84410cf94ef9759 with peer 43ce4a4c954f4a5f80954b38933c189d
I20240501 14:10:35.075984 22871 tablet_copy_client.cc:523] T 85593473500b4d5eaf9f8b0bf2296777 P e6246b183293412494919f1e7efc9448: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:35.079245 22927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db0f01ee72b24ec9b84410cf94ef9759. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:35.081513 22927 tablet_copy_client.cc:783] T db0f01ee72b24ec9b84410cf94ef9759 P e6246b183293412494919f1e7efc9448: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:35.087154 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:35.083498 22871 tablet_bootstrap.cc:492] T 85593473500b4d5eaf9f8b0bf2296777 P e6246b183293412494919f1e7efc9448: Bootstrap starting.
I20240501 14:10:35.088845 22927 tablet_copy_client.cc:647] T db0f01ee72b24ec9b84410cf94ef9759 P e6246b183293412494919f1e7efc9448: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:35.090013 22927 tablet_copy_client.cc:523] T db0f01ee72b24ec9b84410cf94ef9759 P e6246b183293412494919f1e7efc9448: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:35.091164 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:35.091341 22927 tablet_bootstrap.cc:492] T db0f01ee72b24ec9b84410cf94ef9759 P e6246b183293412494919f1e7efc9448: Bootstrap starting.
I20240501 14:10:35.095592 22871 log.cc:824] T 85593473500b4d5eaf9f8b0bf2296777 P e6246b183293412494919f1e7efc9448: Log is configured to *not* fsync() on all Append() calls
I20240501 14:10:35.095646 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:35.100355 19769 consensus_queue.cc:237] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
W20240501 14:10:35.102363 19705 consensus_peers.cc:485] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4 -> Peer 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133:46883): Couldn't send request to peer 5bf81f3d986e428190d3f7727f36d96c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ec2a46f8cfb3406082970e5fdb59b101. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:35.102751 19637 raft_consensus.cc:1261] T ec2a46f8cfb3406082970e5fdb59b101 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:35.103138 22878 consensus_queue.cc:1035] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:35.103873 19376 raft_consensus.cc:1261] T ec2a46f8cfb3406082970e5fdb59b101 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:35.104247 22878 consensus_queue.cc:1035] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:35.113991 22878 raft_consensus.cc:2932] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:35.116317 19375 raft_consensus.cc:2932] T ec2a46f8cfb3406082970e5fdb59b101 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:35.117839 21399 raft_consensus.cc:2932] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:35.119262 19636 raft_consensus.cc:2932] T ec2a46f8cfb3406082970e5fdb59b101 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:35.121618 19226 catalog_manager.cc:5484] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4 reported cstate change: config changed from index -1 to 4, VOTER 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133) added. New cstate: current_term: 1 leader_uuid: "da3c4e948ff747678f18456c134d7ed4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:35.130290 22800 tablet_bootstrap.cc:492] T f69020bb6c244848a6b0a50fd2b4f3eb P 5bf81f3d986e428190d3f7727f36d96c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:35.130647 22800 tablet_bootstrap.cc:492] T f69020bb6c244848a6b0a50fd2b4f3eb P 5bf81f3d986e428190d3f7727f36d96c: Bootstrap complete.
I20240501 14:10:35.130842 22800 ts_tablet_manager.cc:1367] T f69020bb6c244848a6b0a50fd2b4f3eb P 5bf81f3d986e428190d3f7727f36d96c: Time spent bootstrapping tablet: real 0.205s user 0.023s sys 0.006s
I20240501 14:10:35.132550 22800 raft_consensus.cc:348] T f69020bb6c244848a6b0a50fd2b4f3eb P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:35.132719 22800 raft_consensus.cc:729] T f69020bb6c244848a6b0a50fd2b4f3eb P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf81f3d986e428190d3f7727f36d96c, State: Initialized, Role: FOLLOWER
I20240501 14:10:35.132859 22800 consensus_queue.cc:260] T f69020bb6c244848a6b0a50fd2b4f3eb P 5bf81f3d986e428190d3f7727f36d96c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:35.133224 22733 tablet_bootstrap.cc:492] T a9edf1764b55420ea5d519e07ae1d5e0 P 778c78105092428bbcde75ab2cdb6a05: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:35.133632 22733 tablet_bootstrap.cc:492] T a9edf1764b55420ea5d519e07ae1d5e0 P 778c78105092428bbcde75ab2cdb6a05: Bootstrap complete.
I20240501 14:10:35.133818 22733 ts_tablet_manager.cc:1367] T a9edf1764b55420ea5d519e07ae1d5e0 P 778c78105092428bbcde75ab2cdb6a05: Time spent bootstrapping tablet: real 0.086s user 0.022s sys 0.003s
I20240501 14:10:35.134420 22733 raft_consensus.cc:348] T a9edf1764b55420ea5d519e07ae1d5e0 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:10:35.134542 22733 raft_consensus.cc:729] T a9edf1764b55420ea5d519e07ae1d5e0 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 778c78105092428bbcde75ab2cdb6a05, State: Initialized, Role: FOLLOWER
I20240501 14:10:35.134691 22733 consensus_queue.cc:260] T a9edf1764b55420ea5d519e07ae1d5e0 P 778c78105092428bbcde75ab2cdb6a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:10:35.142184 22800 ts_tablet_manager.cc:1398] T f69020bb6c244848a6b0a50fd2b4f3eb P 5bf81f3d986e428190d3f7727f36d96c: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20240501 14:10:35.142717 19396 tablet_copy_service.cc:342] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Request end of tablet copy session 5bf81f3d986e428190d3f7727f36d96c-f69020bb6c244848a6b0a50fd2b4f3eb received from {username='slave'} at 127.7.39.133:53931
I20240501 14:10:35.142808 19396 tablet_copy_service.cc:433] P fb1e6bc2d07b4ce3b622a66f14c93bdc: ending tablet copy session 5bf81f3d986e428190d3f7727f36d96c-f69020bb6c244848a6b0a50fd2b4f3eb on tablet f69020bb6c244848a6b0a50fd2b4f3eb with peer 5bf81f3d986e428190d3f7727f36d96c
I20240501 14:10:35.145041 22985 ts_tablet_manager.cc:902] T 814ba2247a314a4797aa0f9f624aaeb6 P e6246b183293412494919f1e7efc9448: Initiating tablet copy from peer fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847)
I20240501 14:10:35.145612 22985 tablet_copy_client.cc:308] T 814ba2247a314a4797aa0f9f624aaeb6 P e6246b183293412494919f1e7efc9448: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.129:35847
I20240501 14:10:35.146989 22733 ts_tablet_manager.cc:1398] T a9edf1764b55420ea5d519e07ae1d5e0 P 778c78105092428bbcde75ab2cdb6a05: Time spent starting tablet: real 0.013s user 0.001s sys 0.000s
I20240501 14:10:35.149971 19527 tablet_copy_service.cc:342] P 99a0be6ef2144b7fb247ba9d3f051d66: Request end of tablet copy session 778c78105092428bbcde75ab2cdb6a05-a9edf1764b55420ea5d519e07ae1d5e0 received from {username='slave'} at 127.7.39.136:39441
I20240501 14:10:35.150061 19527 tablet_copy_service.cc:433] P 99a0be6ef2144b7fb247ba9d3f051d66: ending tablet copy session 778c78105092428bbcde75ab2cdb6a05-a9edf1764b55420ea5d519e07ae1d5e0 on tablet a9edf1764b55420ea5d519e07ae1d5e0 with peer 778c78105092428bbcde75ab2cdb6a05
I20240501 14:10:35.158228 23017 ts_tablet_manager.cc:902] T 11f64b59f83e458f8cf51fd5bcd36fee P e6246b183293412494919f1e7efc9448: Initiating tablet copy from peer da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357)
I20240501 14:10:35.160513 23017 tablet_copy_client.cc:308] T 11f64b59f83e458f8cf51fd5bcd36fee P e6246b183293412494919f1e7efc9448: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.132:35357
I20240501 14:10:35.163748 19375 consensus_queue.cc:237] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:10:35.165514 19789 tablet_copy_service.cc:140] P da3c4e948ff747678f18456c134d7ed4: Received BeginTabletCopySession request for tablet 11f64b59f83e458f8cf51fd5bcd36fee from peer e6246b183293412494919f1e7efc9448 ({username='slave'} at 127.7.39.134:52289)
I20240501 14:10:35.165547 19506 raft_consensus.cc:1261] T 410a7dd0dd734684a82875912966657d P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:35.165627 19789 tablet_copy_service.cc:161] P da3c4e948ff747678f18456c134d7ed4: Beginning new tablet copy session on tablet 11f64b59f83e458f8cf51fd5bcd36fee from peer e6246b183293412494919f1e7efc9448 at {username='slave'} at 127.7.39.134:52289: session id = e6246b183293412494919f1e7efc9448-11f64b59f83e458f8cf51fd5bcd36fee
I20240501 14:10:35.166402 19789 tablet_copy_source_session.cc:205] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:35.166823 19767 raft_consensus.cc:1261] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:35.167428 22527 consensus_queue.cc:1035] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:35.169224 22527 consensus_queue.cc:1035] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:35.183811 22927 tablet_bootstrap.cc:492] T db0f01ee72b24ec9b84410cf94ef9759 P e6246b183293412494919f1e7efc9448: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:35.184221 22927 tablet_bootstrap.cc:492] T db0f01ee72b24ec9b84410cf94ef9759 P e6246b183293412494919f1e7efc9448: Bootstrap complete.
I20240501 14:10:35.184571 22927 ts_tablet_manager.cc:1367] T db0f01ee72b24ec9b84410cf94ef9759 P e6246b183293412494919f1e7efc9448: Time spent bootstrapping tablet: real 0.093s user 0.028s sys 0.000s
I20240501 14:10:35.184926 22871 tablet_bootstrap.cc:492] T 85593473500b4d5eaf9f8b0bf2296777 P e6246b183293412494919f1e7efc9448: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:35.185216 22871 tablet_bootstrap.cc:492] T 85593473500b4d5eaf9f8b0bf2296777 P e6246b183293412494919f1e7efc9448: Bootstrap complete.
I20240501 14:10:35.185377 22871 ts_tablet_manager.cc:1367] T 85593473500b4d5eaf9f8b0bf2296777 P e6246b183293412494919f1e7efc9448: Time spent bootstrapping tablet: real 0.102s user 0.034s sys 0.000s
I20240501 14:10:35.187892 22927 raft_consensus.cc:348] T db0f01ee72b24ec9b84410cf94ef9759 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:35.187853 22871 raft_consensus.cc:348] T 85593473500b4d5eaf9f8b0bf2296777 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:35.188192 22871 raft_consensus.cc:729] T 85593473500b4d5eaf9f8b0bf2296777 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6246b183293412494919f1e7efc9448, State: Initialized, Role: FOLLOWER
I20240501 14:10:35.188211 22927 raft_consensus.cc:729] T db0f01ee72b24ec9b84410cf94ef9759 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6246b183293412494919f1e7efc9448, State: Initialized, Role: FOLLOWER
I20240501 14:10:35.188709 22927 consensus_queue.cc:260] T db0f01ee72b24ec9b84410cf94ef9759 P e6246b183293412494919f1e7efc9448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:35.188709 22871 consensus_queue.cc:260] T 85593473500b4d5eaf9f8b0bf2296777 P e6246b183293412494919f1e7efc9448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:35.189713 22927 ts_tablet_manager.cc:1398] T db0f01ee72b24ec9b84410cf94ef9759 P e6246b183293412494919f1e7efc9448: Time spent starting tablet: real 0.005s user 0.002s sys 0.004s
I20240501 14:10:35.190852 22871 ts_tablet_manager.cc:1398] T 85593473500b4d5eaf9f8b0bf2296777 P e6246b183293412494919f1e7efc9448: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240501 14:10:35.191682 19658 tablet_copy_service.cc:342] P ec8cde17a00645e7b38d5d29f7425367: Request end of tablet copy session e6246b183293412494919f1e7efc9448-85593473500b4d5eaf9f8b0bf2296777 received from {username='slave'} at 127.7.39.134:41913
I20240501 14:10:35.191769 19658 tablet_copy_service.cc:433] P ec8cde17a00645e7b38d5d29f7425367: ending tablet copy session e6246b183293412494919f1e7efc9448-85593473500b4d5eaf9f8b0bf2296777 on tablet 85593473500b4d5eaf9f8b0bf2296777 with peer e6246b183293412494919f1e7efc9448
I20240501 14:10:35.192080 19657 tablet_copy_service.cc:342] P ec8cde17a00645e7b38d5d29f7425367: Request end of tablet copy session e6246b183293412494919f1e7efc9448-db0f01ee72b24ec9b84410cf94ef9759 received from {username='slave'} at 127.7.39.134:41913
I20240501 14:10:35.192152 19657 tablet_copy_service.cc:433] P ec8cde17a00645e7b38d5d29f7425367: ending tablet copy session e6246b183293412494919f1e7efc9448-db0f01ee72b24ec9b84410cf94ef9759 on tablet db0f01ee72b24ec9b84410cf94ef9759 with peer e6246b183293412494919f1e7efc9448
I20240501 14:10:35.204156 23017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11f64b59f83e458f8cf51fd5bcd36fee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:35.208964 23017 tablet_copy_client.cc:783] T 11f64b59f83e458f8cf51fd5bcd36fee P e6246b183293412494919f1e7efc9448: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:35.209228 23017 tablet_copy_client.cc:647] T 11f64b59f83e458f8cf51fd5bcd36fee P e6246b183293412494919f1e7efc9448: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:35.220145 23017 tablet_copy_client.cc:523] T 11f64b59f83e458f8cf51fd5bcd36fee P e6246b183293412494919f1e7efc9448: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:35.221513 23017 tablet_bootstrap.cc:492] T 11f64b59f83e458f8cf51fd5bcd36fee P e6246b183293412494919f1e7efc9448: Bootstrap starting.
I20240501 14:10:35.222218 19396 tablet_copy_service.cc:140] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Received BeginTabletCopySession request for tablet 814ba2247a314a4797aa0f9f624aaeb6 from peer e6246b183293412494919f1e7efc9448 ({username='slave'} at 127.7.39.134:34039)
I20240501 14:10:35.222337 19396 tablet_copy_service.cc:161] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Beginning new tablet copy session on tablet 814ba2247a314a4797aa0f9f624aaeb6 from peer e6246b183293412494919f1e7efc9448 at {username='slave'} at 127.7.39.134:34039: session id = e6246b183293412494919f1e7efc9448-814ba2247a314a4797aa0f9f624aaeb6
I20240501 14:10:35.223446 19396 tablet_copy_source_session.cc:205] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:35.224279 22985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 814ba2247a314a4797aa0f9f624aaeb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:35.225418 22527 raft_consensus.cc:2932] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:35.226261 22985 tablet_copy_client.cc:783] T 814ba2247a314a4797aa0f9f624aaeb6 P e6246b183293412494919f1e7efc9448: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:35.226377 22985 tablet_copy_client.cc:647] T 814ba2247a314a4797aa0f9f624aaeb6 P e6246b183293412494919f1e7efc9448: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:35.228374 19231 catalog_manager.cc:5484] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc reported cstate change: config changed from index -1 to 4, VOTER 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136) added. New cstate: current_term: 1 leader_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:35.229097 19506 raft_consensus.cc:2932] T 410a7dd0dd734684a82875912966657d P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:35.231750 19767 raft_consensus.cc:2932] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:35.234680 22985 tablet_copy_client.cc:523] T 814ba2247a314a4797aa0f9f624aaeb6 P e6246b183293412494919f1e7efc9448: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:35.236027 22985 tablet_bootstrap.cc:492] T 814ba2247a314a4797aa0f9f624aaeb6 P e6246b183293412494919f1e7efc9448: Bootstrap starting.
I20240501 14:10:35.239570 21399 raft_consensus.cc:1203] T b6be9a115fe14c098c29cad819849f95 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:10:35.241869 21399 raft_consensus.cc:1203] T db0f01ee72b24ec9b84410cf94ef9759 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
W20240501 14:10:35.243927 19309 consensus_peers.cc:485] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136:34895): Couldn't send request to peer 778c78105092428bbcde75ab2cdb6a05. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 410a7dd0dd734684a82875912966657d. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:35.263947 19506 consensus_queue.cc:237] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:10:35.265525 19767 raft_consensus.cc:1261] T 272eee3450cf4f48a4e1de2dcb28c0f7 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Refusing update from remote peer 99a0be6ef2144b7fb247ba9d3f051d66: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:35.265620 19636 raft_consensus.cc:1261] T 272eee3450cf4f48a4e1de2dcb28c0f7 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Refusing update from remote peer 99a0be6ef2144b7fb247ba9d3f051d66: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:35.265939 22081 consensus_queue.cc:1035] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:10:35.266350 19440 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136:34895): Couldn't send request to peer 778c78105092428bbcde75ab2cdb6a05. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 272eee3450cf4f48a4e1de2dcb28c0f7. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:35.266789 22081 consensus_queue.cc:1035] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:35.268702 21530 raft_consensus.cc:1203] T cf62635e9a994259b9e293e5eb425b55 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:10:35.269822 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:35.275528 21137 raft_consensus.cc:1203] T 0988c468757d49738765bdcb31794bd1 P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:10:35.279830 22733 ts_tablet_manager.cc:902] T 55830424a98348e382759c54afdeabcd P 778c78105092428bbcde75ab2cdb6a05: Initiating tablet copy from peer 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829)
I20240501 14:10:35.280295 22733 tablet_copy_client.cc:308] T 55830424a98348e382759c54afdeabcd P 778c78105092428bbcde75ab2cdb6a05: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.130:41829
I20240501 14:10:35.281270 19527 tablet_copy_service.cc:140] P 99a0be6ef2144b7fb247ba9d3f051d66: Received BeginTabletCopySession request for tablet 55830424a98348e382759c54afdeabcd from peer 778c78105092428bbcde75ab2cdb6a05 ({username='slave'} at 127.7.39.136:39441)
I20240501 14:10:35.281365 19527 tablet_copy_service.cc:161] P 99a0be6ef2144b7fb247ba9d3f051d66: Beginning new tablet copy session on tablet 55830424a98348e382759c54afdeabcd from peer 778c78105092428bbcde75ab2cdb6a05 at {username='slave'} at 127.7.39.136:39441: session id = 778c78105092428bbcde75ab2cdb6a05-55830424a98348e382759c54afdeabcd
I20240501 14:10:35.282256 19527 tablet_copy_source_session.cc:205] T 55830424a98348e382759c54afdeabcd P 99a0be6ef2144b7fb247ba9d3f051d66: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:35.282452 22985 tablet_bootstrap.cc:492] T 814ba2247a314a4797aa0f9f624aaeb6 P e6246b183293412494919f1e7efc9448: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:35.282757 22985 tablet_bootstrap.cc:492] T 814ba2247a314a4797aa0f9f624aaeb6 P e6246b183293412494919f1e7efc9448: Bootstrap complete.
I20240501 14:10:35.283004 22985 ts_tablet_manager.cc:1367] T 814ba2247a314a4797aa0f9f624aaeb6 P e6246b183293412494919f1e7efc9448: Time spent bootstrapping tablet: real 0.047s user 0.027s sys 0.000s
I20240501 14:10:35.283596 22985 raft_consensus.cc:348] T 814ba2247a314a4797aa0f9f624aaeb6 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:35.283811 22985 raft_consensus.cc:729] T 814ba2247a314a4797aa0f9f624aaeb6 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6246b183293412494919f1e7efc9448, State: Initialized, Role: FOLLOWER
I20240501 14:10:35.284039 22985 consensus_queue.cc:260] T 814ba2247a314a4797aa0f9f624aaeb6 P e6246b183293412494919f1e7efc9448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:35.285813 22985 ts_tablet_manager.cc:1398] T 814ba2247a314a4797aa0f9f624aaeb6 P e6246b183293412494919f1e7efc9448: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:10:35.286341 19396 tablet_copy_service.cc:342] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Request end of tablet copy session e6246b183293412494919f1e7efc9448-814ba2247a314a4797aa0f9f624aaeb6 received from {username='slave'} at 127.7.39.134:34039
I20240501 14:10:35.286439 19396 tablet_copy_service.cc:433] P fb1e6bc2d07b4ce3b622a66f14c93bdc: ending tablet copy session e6246b183293412494919f1e7efc9448-814ba2247a314a4797aa0f9f624aaeb6 on tablet 814ba2247a314a4797aa0f9f624aaeb6 with peer e6246b183293412494919f1e7efc9448
I20240501 14:10:35.287638 22733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55830424a98348e382759c54afdeabcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:35.291916 22733 tablet_copy_client.cc:783] T 55830424a98348e382759c54afdeabcd P 778c78105092428bbcde75ab2cdb6a05: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:35.292099 22733 tablet_copy_client.cc:647] T 55830424a98348e382759c54afdeabcd P 778c78105092428bbcde75ab2cdb6a05: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:35.294281 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:35.300406 21399 raft_consensus.cc:1203] T ad7c255baee74e629e2976a942a4174f P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:10:35.305485 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:35.311242 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:35.312708 22733 tablet_copy_client.cc:523] T 55830424a98348e382759c54afdeabcd P 778c78105092428bbcde75ab2cdb6a05: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:35.315721 22733 tablet_bootstrap.cc:492] T 55830424a98348e382759c54afdeabcd P 778c78105092428bbcde75ab2cdb6a05: Bootstrap starting.
I20240501 14:10:35.315878 23017 tablet_bootstrap.cc:492] T 11f64b59f83e458f8cf51fd5bcd36fee P e6246b183293412494919f1e7efc9448: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:35.316272 23017 tablet_bootstrap.cc:492] T 11f64b59f83e458f8cf51fd5bcd36fee P e6246b183293412494919f1e7efc9448: Bootstrap complete.
I20240501 14:10:35.316529 23017 ts_tablet_manager.cc:1367] T 11f64b59f83e458f8cf51fd5bcd36fee P e6246b183293412494919f1e7efc9448: Time spent bootstrapping tablet: real 0.095s user 0.030s sys 0.000s
I20240501 14:10:35.316990 23017 raft_consensus.cc:348] T 11f64b59f83e458f8cf51fd5bcd36fee P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:35.317203 23017 raft_consensus.cc:729] T 11f64b59f83e458f8cf51fd5bcd36fee P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6246b183293412494919f1e7efc9448, State: Initialized, Role: FOLLOWER
I20240501 14:10:35.317421 23017 consensus_queue.cc:260] T 11f64b59f83e458f8cf51fd5bcd36fee P e6246b183293412494919f1e7efc9448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:35.317862 23017 ts_tablet_manager.cc:1398] T 11f64b59f83e458f8cf51fd5bcd36fee P e6246b183293412494919f1e7efc9448: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:35.318421 19789 tablet_copy_service.cc:342] P da3c4e948ff747678f18456c134d7ed4: Request end of tablet copy session e6246b183293412494919f1e7efc9448-11f64b59f83e458f8cf51fd5bcd36fee received from {username='slave'} at 127.7.39.134:52289
I20240501 14:10:35.318507 19789 tablet_copy_service.cc:433] P da3c4e948ff747678f18456c134d7ed4: ending tablet copy session e6246b183293412494919f1e7efc9448-11f64b59f83e458f8cf51fd5bcd36fee on tablet 11f64b59f83e458f8cf51fd5bcd36fee with peer e6246b183293412494919f1e7efc9448
I20240501 14:10:35.326462 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:35.327039 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:35.328688 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:35.329731 21137 raft_consensus.cc:1203] T f69020bb6c244848a6b0a50fd2b4f3eb P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:10:35.334271 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:35.335009 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:35.338688 21137 raft_consensus.cc:1203] T 7eea58d4a33f422a9ada1e82d408577f P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:10:35.345245 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:35.357620 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:35.362159 23096 raft_consensus.cc:2932] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:35.364959 19767 raft_consensus.cc:2932] T 272eee3450cf4f48a4e1de2dcb28c0f7 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:35.365653 19228 catalog_manager.cc:5484] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 reported cstate change: config changed from index -1 to 4, VOTER 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136) added. New cstate: current_term: 1 leader_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:35.368197 19637 raft_consensus.cc:2932] T 272eee3450cf4f48a4e1de2dcb28c0f7 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:35.368429 19636 consensus_queue.cc:237] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
W20240501 14:10:35.370772 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2dcca6e0e6374520bd4522c80c110e97. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:35.370900 19376 raft_consensus.cc:1261] T 2dcca6e0e6374520bd4522c80c110e97 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:35.371039 19506 raft_consensus.cc:1261] T 2dcca6e0e6374520bd4522c80c110e97 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:35.371340 20654 consensus_queue.cc:1035] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:35.372294 20654 consensus_queue.cc:1035] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:35.377478 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:35.378576 21268 raft_consensus.cc:1203] T 85593473500b4d5eaf9f8b0bf2296777 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:10:35.393018 21268 raft_consensus.cc:1203] T db0f01ee72b24ec9b84410cf94ef9759 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:10:35.396678 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:35.400949 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:35.410703 21399 raft_consensus.cc:1203] T 7eea58d4a33f422a9ada1e82d408577f P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:10:35.419193 21248 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:35.422293 22838 ts_tablet_manager.cc:902] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 43ce4a4c954f4a5f80954b38933c189d: Initiating tablet copy from peer ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201)
I20240501 14:10:35.422744 22838 tablet_copy_client.cc:308] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.131:45201
I20240501 14:10:35.423224 19657 tablet_copy_service.cc:140] P ec8cde17a00645e7b38d5d29f7425367: Received BeginTabletCopySession request for tablet 10aa2feeb0324aa4a6221d2f2f3fc941 from peer 43ce4a4c954f4a5f80954b38933c189d ({username='slave'} at 127.7.39.135:35759)
I20240501 14:10:35.423347 19657 tablet_copy_service.cc:161] P ec8cde17a00645e7b38d5d29f7425367: Beginning new tablet copy session on tablet 10aa2feeb0324aa4a6221d2f2f3fc941 from peer 43ce4a4c954f4a5f80954b38933c189d at {username='slave'} at 127.7.39.135:35759: session id = 43ce4a4c954f4a5f80954b38933c189d-10aa2feeb0324aa4a6221d2f2f3fc941
I20240501 14:10:35.424371 19657 tablet_copy_source_session.cc:205] T 10aa2feeb0324aa4a6221d2f2f3fc941 P ec8cde17a00645e7b38d5d29f7425367: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:35.425312 22838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10aa2feeb0324aa4a6221d2f2f3fc941. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:35.428048 22838 tablet_copy_client.cc:783] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:35.428277 22838 tablet_copy_client.cc:647] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:35.429476 22838 tablet_copy_client.cc:523] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:35.430728 22838 tablet_bootstrap.cc:492] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 43ce4a4c954f4a5f80954b38933c189d: Bootstrap starting.
I20240501 14:10:35.431499 22733 tablet_bootstrap.cc:492] T 55830424a98348e382759c54afdeabcd P 778c78105092428bbcde75ab2cdb6a05: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:35.431823 22733 tablet_bootstrap.cc:492] T 55830424a98348e382759c54afdeabcd P 778c78105092428bbcde75ab2cdb6a05: Bootstrap complete.
I20240501 14:10:35.432031 22733 ts_tablet_manager.cc:1367] T 55830424a98348e382759c54afdeabcd P 778c78105092428bbcde75ab2cdb6a05: Time spent bootstrapping tablet: real 0.116s user 0.025s sys 0.007s
I20240501 14:10:35.432569 21248 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:35.432660 22733 raft_consensus.cc:348] T 55830424a98348e382759c54afdeabcd P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:10:35.432816 22733 raft_consensus.cc:729] T 55830424a98348e382759c54afdeabcd P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 778c78105092428bbcde75ab2cdb6a05, State: Initialized, Role: FOLLOWER
I20240501 14:10:35.432965 22733 consensus_queue.cc:260] T 55830424a98348e382759c54afdeabcd P 778c78105092428bbcde75ab2cdb6a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:10:35.433336 22733 ts_tablet_manager.cc:1398] T 55830424a98348e382759c54afdeabcd P 778c78105092428bbcde75ab2cdb6a05: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:35.434913 19527 tablet_copy_service.cc:342] P 99a0be6ef2144b7fb247ba9d3f051d66: Request end of tablet copy session 778c78105092428bbcde75ab2cdb6a05-55830424a98348e382759c54afdeabcd received from {username='slave'} at 127.7.39.136:39441
I20240501 14:10:35.435017 19527 tablet_copy_service.cc:433] P 99a0be6ef2144b7fb247ba9d3f051d66: ending tablet copy session 778c78105092428bbcde75ab2cdb6a05-55830424a98348e382759c54afdeabcd on tablet 55830424a98348e382759c54afdeabcd with peer 778c78105092428bbcde75ab2cdb6a05
I20240501 14:10:35.444500 20654 raft_consensus.cc:2932] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:35.446971 19506 raft_consensus.cc:2932] T 2dcca6e0e6374520bd4522c80c110e97 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:35.448114 19376 raft_consensus.cc:2932] T 2dcca6e0e6374520bd4522c80c110e97 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:35.451828 21530 raft_consensus.cc:1203] T a9edf1764b55420ea5d519e07ae1d5e0 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:10:35.455664 19232 catalog_manager.cc:5484] T 2dcca6e0e6374520bd4522c80c110e97 P 99a0be6ef2144b7fb247ba9d3f051d66 reported cstate change: config changed from index -1 to 4, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:35.460834 22838 tablet_bootstrap.cc:492] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 43ce4a4c954f4a5f80954b38933c189d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:35.461439 22838 tablet_bootstrap.cc:492] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 43ce4a4c954f4a5f80954b38933c189d: Bootstrap complete.
I20240501 14:10:35.461634 22838 ts_tablet_manager.cc:1367] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 43ce4a4c954f4a5f80954b38933c189d: Time spent bootstrapping tablet: real 0.031s user 0.024s sys 0.000s
I20240501 14:10:35.462123 22838 raft_consensus.cc:348] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:35.462242 22838 raft_consensus.cc:729] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43ce4a4c954f4a5f80954b38933c189d, State: Initialized, Role: FOLLOWER
I20240501 14:10:35.462430 22838 consensus_queue.cc:260] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 43ce4a4c954f4a5f80954b38933c189d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:35.463593 22838 ts_tablet_manager.cc:1398] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 43ce4a4c954f4a5f80954b38933c189d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:10:35.464109 19657 tablet_copy_service.cc:342] P ec8cde17a00645e7b38d5d29f7425367: Request end of tablet copy session 43ce4a4c954f4a5f80954b38933c189d-10aa2feeb0324aa4a6221d2f2f3fc941 received from {username='slave'} at 127.7.39.135:35759
I20240501 14:10:35.464190 19657 tablet_copy_service.cc:433] P ec8cde17a00645e7b38d5d29f7425367: ending tablet copy session 43ce4a4c954f4a5f80954b38933c189d-10aa2feeb0324aa4a6221d2f2f3fc941 on tablet 10aa2feeb0324aa4a6221d2f2f3fc941 with peer 43ce4a4c954f4a5f80954b38933c189d
I20240501 14:10:35.473268 22733 ts_tablet_manager.cc:902] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 778c78105092428bbcde75ab2cdb6a05: Initiating tablet copy from peer ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201)
I20240501 14:10:35.473737 22733 tablet_copy_client.cc:308] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 778c78105092428bbcde75ab2cdb6a05: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.131:45201
I20240501 14:10:35.483742 19657 tablet_copy_service.cc:140] P ec8cde17a00645e7b38d5d29f7425367: Received BeginTabletCopySession request for tablet 10aa2feeb0324aa4a6221d2f2f3fc941 from peer 778c78105092428bbcde75ab2cdb6a05 ({username='slave'} at 127.7.39.136:34889)
I20240501 14:10:35.483871 19657 tablet_copy_service.cc:161] P ec8cde17a00645e7b38d5d29f7425367: Beginning new tablet copy session on tablet 10aa2feeb0324aa4a6221d2f2f3fc941 from peer 778c78105092428bbcde75ab2cdb6a05 at {username='slave'} at 127.7.39.136:34889: session id = 778c78105092428bbcde75ab2cdb6a05-10aa2feeb0324aa4a6221d2f2f3fc941
I20240501 14:10:35.484797 19657 tablet_copy_source_session.cc:205] T 10aa2feeb0324aa4a6221d2f2f3fc941 P ec8cde17a00645e7b38d5d29f7425367: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:35.485934 22733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10aa2feeb0324aa4a6221d2f2f3fc941. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:35.488366 22733 tablet_copy_client.cc:783] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 778c78105092428bbcde75ab2cdb6a05: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:35.488552 22733 tablet_copy_client.cc:647] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 778c78105092428bbcde75ab2cdb6a05: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:35.493863 22733 tablet_copy_client.cc:523] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 778c78105092428bbcde75ab2cdb6a05: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:35.495224 22733 tablet_bootstrap.cc:492] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 778c78105092428bbcde75ab2cdb6a05: Bootstrap starting.
I20240501 14:10:35.500586 22800 ts_tablet_manager.cc:902] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 5bf81f3d986e428190d3f7727f36d96c: Initiating tablet copy from peer fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847)
I20240501 14:10:35.501019 22800 tablet_copy_client.cc:308] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 5bf81f3d986e428190d3f7727f36d96c: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.129:35847
I20240501 14:10:35.501613 19396 tablet_copy_service.cc:140] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Received BeginTabletCopySession request for tablet 98af8b06b9324e3fa7cdb898b7b9fab9 from peer 5bf81f3d986e428190d3f7727f36d96c ({username='slave'} at 127.7.39.133:53931)
I20240501 14:10:35.501725 19396 tablet_copy_service.cc:161] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Beginning new tablet copy session on tablet 98af8b06b9324e3fa7cdb898b7b9fab9 from peer 5bf81f3d986e428190d3f7727f36d96c at {username='slave'} at 127.7.39.133:53931: session id = 5bf81f3d986e428190d3f7727f36d96c-98af8b06b9324e3fa7cdb898b7b9fab9
I20240501 14:10:35.502568 19767 consensus_queue.cc:237] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:35.502655 19396 tablet_copy_source_session.cc:205] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:35.503814 22800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98af8b06b9324e3fa7cdb898b7b9fab9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:35.504462 19376 raft_consensus.cc:1261] T 1d408b075d784953a57554bcdf19158a P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:35.504590 19506 raft_consensus.cc:1261] T 1d408b075d784953a57554bcdf19158a P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:35.504876 22877 consensus_queue.cc:1035] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:35.505034 22411 consensus_queue.cc:1035] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:10:35.506760 19705 consensus_peers.cc:485] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4 -> Peer 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133:46883): Couldn't send request to peer 5bf81f3d986e428190d3f7727f36d96c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1d408b075d784953a57554bcdf19158a. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:35.507894 22800 tablet_copy_client.cc:783] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 5bf81f3d986e428190d3f7727f36d96c: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:35.508070 22800 tablet_copy_client.cc:647] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 5bf81f3d986e428190d3f7727f36d96c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:35.511865 22800 tablet_copy_client.cc:523] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 5bf81f3d986e428190d3f7727f36d96c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:35.514145 22800 tablet_bootstrap.cc:492] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 5bf81f3d986e428190d3f7727f36d96c: Bootstrap starting.
I20240501 14:10:35.515697 22877 raft_consensus.cc:2932] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:35.517987 19376 raft_consensus.cc:2932] T 1d408b075d784953a57554bcdf19158a P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:35.520665 19233 catalog_manager.cc:5484] T 1d408b075d784953a57554bcdf19158a P fb1e6bc2d07b4ce3b622a66f14c93bdc reported cstate change: config changed from index -1 to 4, VOTER 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133) added. New cstate: current_term: 1 leader_uuid: "da3c4e948ff747678f18456c134d7ed4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:35.522192 19506 raft_consensus.cc:2932] T 1d408b075d784953a57554bcdf19158a P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:35.540422 22733 tablet_bootstrap.cc:492] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 778c78105092428bbcde75ab2cdb6a05: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:35.540758 22733 tablet_bootstrap.cc:492] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 778c78105092428bbcde75ab2cdb6a05: Bootstrap complete.
I20240501 14:10:35.540953 22733 ts_tablet_manager.cc:1367] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 778c78105092428bbcde75ab2cdb6a05: Time spent bootstrapping tablet: real 0.046s user 0.028s sys 0.000s
I20240501 14:10:35.541432 22733 raft_consensus.cc:348] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:35.541564 22733 raft_consensus.cc:729] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 778c78105092428bbcde75ab2cdb6a05, State: Initialized, Role: FOLLOWER
I20240501 14:10:35.541693 22733 consensus_queue.cc:260] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 778c78105092428bbcde75ab2cdb6a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:35.542062 22733 ts_tablet_manager.cc:1398] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 778c78105092428bbcde75ab2cdb6a05: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:10:35.542886 19657 tablet_copy_service.cc:342] P ec8cde17a00645e7b38d5d29f7425367: Request end of tablet copy session 778c78105092428bbcde75ab2cdb6a05-10aa2feeb0324aa4a6221d2f2f3fc941 received from {username='slave'} at 127.7.39.136:34889
I20240501 14:10:35.542985 19657 tablet_copy_service.cc:433] P ec8cde17a00645e7b38d5d29f7425367: ending tablet copy session 778c78105092428bbcde75ab2cdb6a05-10aa2feeb0324aa4a6221d2f2f3fc941 on tablet 10aa2feeb0324aa4a6221d2f2f3fc941 with peer 778c78105092428bbcde75ab2cdb6a05
I20240501 14:10:35.543493 19637 consensus_queue.cc:237] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:35.545903 19506 raft_consensus.cc:1261] T cc8e888d46d34a22b6d4409ec2ab53a9 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:10:35.546555 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cc8e888d46d34a22b6d4409ec2ab53a9. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:35.546672 22086 consensus_queue.cc:1035] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:35.549105 19376 raft_consensus.cc:1261] T cc8e888d46d34a22b6d4409ec2ab53a9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:35.550781 22458 consensus_queue.cc:1035] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:35.564103 22086 raft_consensus.cc:2932] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:35.566602 19506 raft_consensus.cc:2932] T cc8e888d46d34a22b6d4409ec2ab53a9 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:35.566233 19376 raft_consensus.cc:2932] T cc8e888d46d34a22b6d4409ec2ab53a9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:35.569165 19233 catalog_manager.cc:5484] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 reported cstate change: config changed from index -1 to 4, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:35.587618 22800 tablet_bootstrap.cc:492] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 5bf81f3d986e428190d3f7727f36d96c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:35.587970 22800 tablet_bootstrap.cc:492] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 5bf81f3d986e428190d3f7727f36d96c: Bootstrap complete.
I20240501 14:10:35.588227 22800 ts_tablet_manager.cc:1367] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 5bf81f3d986e428190d3f7727f36d96c: Time spent bootstrapping tablet: real 0.074s user 0.027s sys 0.003s
I20240501 14:10:35.588842 22800 raft_consensus.cc:348] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:35.589008 22800 raft_consensus.cc:729] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf81f3d986e428190d3f7727f36d96c, State: Initialized, Role: FOLLOWER
I20240501 14:10:35.589257 22800 consensus_queue.cc:260] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 5bf81f3d986e428190d3f7727f36d96c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:35.590412 22800 ts_tablet_manager.cc:1398] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 5bf81f3d986e428190d3f7727f36d96c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:10:35.590909 19396 tablet_copy_service.cc:342] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Request end of tablet copy session 5bf81f3d986e428190d3f7727f36d96c-98af8b06b9324e3fa7cdb898b7b9fab9 received from {username='slave'} at 127.7.39.133:53931
I20240501 14:10:35.591006 19396 tablet_copy_service.cc:433] P fb1e6bc2d07b4ce3b622a66f14c93bdc: ending tablet copy session 5bf81f3d986e428190d3f7727f36d96c-98af8b06b9324e3fa7cdb898b7b9fab9 on tablet 98af8b06b9324e3fa7cdb898b7b9fab9 with peer 5bf81f3d986e428190d3f7727f36d96c
I20240501 14:10:35.593875 22800 ts_tablet_manager.cc:902] T ec2a46f8cfb3406082970e5fdb59b101 P 5bf81f3d986e428190d3f7727f36d96c: Initiating tablet copy from peer da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357)
I20240501 14:10:35.594252 22800 tablet_copy_client.cc:308] T ec2a46f8cfb3406082970e5fdb59b101 P 5bf81f3d986e428190d3f7727f36d96c: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.132:35357
I20240501 14:10:35.594718 19789 tablet_copy_service.cc:140] P da3c4e948ff747678f18456c134d7ed4: Received BeginTabletCopySession request for tablet ec2a46f8cfb3406082970e5fdb59b101 from peer 5bf81f3d986e428190d3f7727f36d96c ({username='slave'} at 127.7.39.133:51567)
I20240501 14:10:35.594820 19789 tablet_copy_service.cc:161] P da3c4e948ff747678f18456c134d7ed4: Beginning new tablet copy session on tablet ec2a46f8cfb3406082970e5fdb59b101 from peer 5bf81f3d986e428190d3f7727f36d96c at {username='slave'} at 127.7.39.133:51567: session id = 5bf81f3d986e428190d3f7727f36d96c-ec2a46f8cfb3406082970e5fdb59b101
I20240501 14:10:35.595969 19789 tablet_copy_source_session.cc:205] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:35.596897 22800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec2a46f8cfb3406082970e5fdb59b101. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:35.599427 22800 tablet_copy_client.cc:783] T ec2a46f8cfb3406082970e5fdb59b101 P 5bf81f3d986e428190d3f7727f36d96c: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:35.599593 22800 tablet_copy_client.cc:647] T ec2a46f8cfb3406082970e5fdb59b101 P 5bf81f3d986e428190d3f7727f36d96c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:35.601034 22800 tablet_copy_client.cc:523] T ec2a46f8cfb3406082970e5fdb59b101 P 5bf81f3d986e428190d3f7727f36d96c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:35.602550 22800 tablet_bootstrap.cc:492] T ec2a46f8cfb3406082970e5fdb59b101 P 5bf81f3d986e428190d3f7727f36d96c: Bootstrap starting.
I20240501 14:10:35.603716 19637 consensus_queue.cc:237] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:35.605973 19506 raft_consensus.cc:1261] T fe554c3145de46cf8070efcecffbaedb P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:35.607156 19769 raft_consensus.cc:1261] T fe554c3145de46cf8070efcecffbaedb P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:35.607364 19376 consensus_queue.cc:237] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:35.607570 22458 consensus_queue.cc:1035] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:10:35.608323 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fe554c3145de46cf8070efcecffbaedb. This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:35.609850 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d63afd081986473aa2715d074b123a93. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:35.611646 19769 raft_consensus.cc:1261] T d63afd081986473aa2715d074b123a93 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:35.612084 22526 consensus_queue.cc:1035] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:35.613026 19506 raft_consensus.cc:1261] T d63afd081986473aa2715d074b123a93 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:35.613446 22458 consensus_queue.cc:1035] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:35.614120 22526 consensus_queue.cc:1035] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:35.617544 22458 raft_consensus.cc:2932] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:35.620116 19767 raft_consensus.cc:2932] T fe554c3145de46cf8070efcecffbaedb P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:35.620724 19225 catalog_manager.cc:5484] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 reported cstate change: config changed from index -1 to 4, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:35.623498 19506 raft_consensus.cc:2932] T fe554c3145de46cf8070efcecffbaedb P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:35.626525 22526 raft_consensus.cc:2932] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:35.628947 19769 raft_consensus.cc:2932] T d63afd081986473aa2715d074b123a93 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:35.629825 19505 raft_consensus.cc:2932] T d63afd081986473aa2715d074b123a93 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:35.617620 19637 consensus_queue.cc:237] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:35.634194 22800 tablet_bootstrap.cc:492] T ec2a46f8cfb3406082970e5fdb59b101 P 5bf81f3d986e428190d3f7727f36d96c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:35.634497 22800 tablet_bootstrap.cc:492] T ec2a46f8cfb3406082970e5fdb59b101 P 5bf81f3d986e428190d3f7727f36d96c: Bootstrap complete.
I20240501 14:10:35.634672 22800 ts_tablet_manager.cc:1367] T ec2a46f8cfb3406082970e5fdb59b101 P 5bf81f3d986e428190d3f7727f36d96c: Time spent bootstrapping tablet: real 0.032s user 0.026s sys 0.000s
I20240501 14:10:35.635182 22800 raft_consensus.cc:348] T ec2a46f8cfb3406082970e5fdb59b101 P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:35.635301 22800 raft_consensus.cc:729] T ec2a46f8cfb3406082970e5fdb59b101 P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf81f3d986e428190d3f7727f36d96c, State: Initialized, Role: FOLLOWER
I20240501 14:10:35.635427 22800 consensus_queue.cc:260] T ec2a46f8cfb3406082970e5fdb59b101 P 5bf81f3d986e428190d3f7727f36d96c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:35.636250 22800 ts_tablet_manager.cc:1398] T ec2a46f8cfb3406082970e5fdb59b101 P 5bf81f3d986e428190d3f7727f36d96c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 14:10:35.636652 19376 raft_consensus.cc:1261] T 2dcca6e0e6374520bd4522c80c110e97 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:35.637017 19789 tablet_copy_service.cc:342] P da3c4e948ff747678f18456c134d7ed4: Request end of tablet copy session 5bf81f3d986e428190d3f7727f36d96c-ec2a46f8cfb3406082970e5fdb59b101 received from {username='slave'} at 127.7.39.133:51567
I20240501 14:10:35.637092 19789 tablet_copy_service.cc:433] P da3c4e948ff747678f18456c134d7ed4: ending tablet copy session 5bf81f3d986e428190d3f7727f36d96c-ec2a46f8cfb3406082970e5fdb59b101 on tablet ec2a46f8cfb3406082970e5fdb59b101 with peer 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:35.637071 19571 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135:38751): Couldn't send request to peer 43ce4a4c954f4a5f80954b38933c189d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2dcca6e0e6374520bd4522c80c110e97. This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:35.637792 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2dcca6e0e6374520bd4522c80c110e97. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:35.638119 22086 consensus_queue.cc:1035] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:35.639989 19506 raft_consensus.cc:1261] T 2dcca6e0e6374520bd4522c80c110e97 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:35.640166 19225 catalog_manager.cc:5484] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc reported cstate change: config changed from index -1 to 4, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New cstate: current_term: 1 leader_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:35.640461 22086 consensus_queue.cc:1035] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:35.645771 22733 ts_tablet_manager.cc:902] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 778c78105092428bbcde75ab2cdb6a05: Initiating tablet copy from peer 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829)
I20240501 14:10:35.646283 22733 tablet_copy_client.cc:308] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 778c78105092428bbcde75ab2cdb6a05: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.130:41829
I20240501 14:10:35.647120 22086 raft_consensus.cc:2932] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:35.648103 19527 tablet_copy_service.cc:140] P 99a0be6ef2144b7fb247ba9d3f051d66: Received BeginTabletCopySession request for tablet 272eee3450cf4f48a4e1de2dcb28c0f7 from peer 778c78105092428bbcde75ab2cdb6a05 ({username='slave'} at 127.7.39.136:39441)
I20240501 14:10:35.648218 19527 tablet_copy_service.cc:161] P 99a0be6ef2144b7fb247ba9d3f051d66: Beginning new tablet copy session on tablet 272eee3450cf4f48a4e1de2dcb28c0f7 from peer 778c78105092428bbcde75ab2cdb6a05 at {username='slave'} at 127.7.39.136:39441: session id = 778c78105092428bbcde75ab2cdb6a05-272eee3450cf4f48a4e1de2dcb28c0f7
I20240501 14:10:35.649341 19527 tablet_copy_source_session.cc:205] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:35.650519 22733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 272eee3450cf4f48a4e1de2dcb28c0f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:35.650801 19231 catalog_manager.cc:5484] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 reported cstate change: config changed from index 4 to 5, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:35.651639 19506 raft_consensus.cc:2932] T 2dcca6e0e6374520bd4522c80c110e97 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:35.652009 19376 raft_consensus.cc:2932] T 2dcca6e0e6374520bd4522c80c110e97 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:35.652842 22733 tablet_copy_client.cc:783] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 778c78105092428bbcde75ab2cdb6a05: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:35.653005 22733 tablet_copy_client.cc:647] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 778c78105092428bbcde75ab2cdb6a05: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:35.669116 22733 tablet_copy_client.cc:523] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 778c78105092428bbcde75ab2cdb6a05: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:35.670866 22733 tablet_bootstrap.cc:492] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 778c78105092428bbcde75ab2cdb6a05: Bootstrap starting.
I20240501 14:10:35.681233 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:35.683362 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:35.689013 23341 ts_tablet_manager.cc:902] T 410a7dd0dd734684a82875912966657d P 778c78105092428bbcde75ab2cdb6a05: Initiating tablet copy from peer fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847)
I20240501 14:10:35.689734 23341 tablet_copy_client.cc:308] T 410a7dd0dd734684a82875912966657d P 778c78105092428bbcde75ab2cdb6a05: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.129:35847
I20240501 14:10:35.697404 19396 tablet_copy_service.cc:140] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Received BeginTabletCopySession request for tablet 410a7dd0dd734684a82875912966657d from peer 778c78105092428bbcde75ab2cdb6a05 ({username='slave'} at 127.7.39.136:56905)
I20240501 14:10:35.697536 19396 tablet_copy_service.cc:161] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Beginning new tablet copy session on tablet 410a7dd0dd734684a82875912966657d from peer 778c78105092428bbcde75ab2cdb6a05 at {username='slave'} at 127.7.39.136:56905: session id = 778c78105092428bbcde75ab2cdb6a05-410a7dd0dd734684a82875912966657d
I20240501 14:10:35.698554 19396 tablet_copy_source_session.cc:205] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:35.699540 23341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 410a7dd0dd734684a82875912966657d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:35.701007 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:35.702880 23341 tablet_copy_client.cc:783] T 410a7dd0dd734684a82875912966657d P 778c78105092428bbcde75ab2cdb6a05: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:35.703119 23341 tablet_copy_client.cc:647] T 410a7dd0dd734684a82875912966657d P 778c78105092428bbcde75ab2cdb6a05: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:35.705338 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:35.713228 21268 raft_consensus.cc:1203] T 814ba2247a314a4797aa0f9f624aaeb6 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:10:35.717821 21248 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:35.718911 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:35.735458 23341 tablet_copy_client.cc:523] T 410a7dd0dd734684a82875912966657d P 778c78105092428bbcde75ab2cdb6a05: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:35.737313 23341 tablet_bootstrap.cc:492] T 410a7dd0dd734684a82875912966657d P 778c78105092428bbcde75ab2cdb6a05: Bootstrap starting.
I20240501 14:10:35.738590 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:35.748188 21268 raft_consensus.cc:1203] T 11f64b59f83e458f8cf51fd5bcd36fee P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:10:35.752444 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:35.756167 19636 consensus_queue.cc:237] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:35.758003 19769 consensus_queue.cc:237] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
W20240501 14:10:35.759176 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cc8e888d46d34a22b6d4409ec2ab53a9. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:35.759730 19505 raft_consensus.cc:1261] T cc8e888d46d34a22b6d4409ec2ab53a9 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:35.759832 19636 raft_consensus.cc:1261] T ec2a46f8cfb3406082970e5fdb59b101 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:35.759888 19376 raft_consensus.cc:1261] T ec2a46f8cfb3406082970e5fdb59b101 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:35.760252 22411 consensus_queue.cc:1035] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:35.760355 22410 consensus_queue.cc:1035] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 14:10:35.761217 19571 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135:38751): Couldn't send request to peer 43ce4a4c954f4a5f80954b38933c189d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cc8e888d46d34a22b6d4409ec2ab53a9. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:35.761358 21137 raft_consensus.cc:1261] T ec2a46f8cfb3406082970e5fdb59b101 P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:35.761598 19375 raft_consensus.cc:1261] T cc8e888d46d34a22b6d4409ec2ab53a9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:35.761890 20654 consensus_queue.cc:1035] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:35.761982 22410 consensus_queue.cc:1035] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 14:10:35.762243 19702 consensus_peers.cc:485] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4 -> Peer 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136:34895): Couldn't send request to peer 778c78105092428bbcde75ab2cdb6a05. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ec2a46f8cfb3406082970e5fdb59b101. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:35.764714 22733 tablet_bootstrap.cc:492] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 778c78105092428bbcde75ab2cdb6a05: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:35.765303 20654 consensus_queue.cc:1035] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:35.765794 22410 raft_consensus.cc:2932] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:35.767334 22733 tablet_bootstrap.cc:492] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 778c78105092428bbcde75ab2cdb6a05: Bootstrap complete.
I20240501 14:10:35.767599 22733 ts_tablet_manager.cc:1367] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 778c78105092428bbcde75ab2cdb6a05: Time spent bootstrapping tablet: real 0.097s user 0.028s sys 0.001s
I20240501 14:10:35.767549 20654 raft_consensus.cc:2932] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:35.768453 19376 raft_consensus.cc:2932] T ec2a46f8cfb3406082970e5fdb59b101 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:35.768918 19231 catalog_manager.cc:5484] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4 reported cstate change: config changed from index 4 to 5, VOTER 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136) added. New cstate: current_term: 1 leader_uuid: "da3c4e948ff747678f18456c134d7ed4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:35.770607 19505 raft_consensus.cc:2932] T cc8e888d46d34a22b6d4409ec2ab53a9 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:35.771278 19232 catalog_manager.cc:5484] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 reported cstate change: config changed from index 4 to 5, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:35.774266 19376 raft_consensus.cc:2932] T cc8e888d46d34a22b6d4409ec2ab53a9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:35.774730 21137 raft_consensus.cc:2932] T ec2a46f8cfb3406082970e5fdb59b101 P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:35.776587 22733 raft_consensus.cc:348] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:10:35.776718 22733 raft_consensus.cc:729] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 778c78105092428bbcde75ab2cdb6a05, State: Initialized, Role: FOLLOWER
I20240501 14:10:35.776908 22733 consensus_queue.cc:260] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 778c78105092428bbcde75ab2cdb6a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:10:35.777760 22733 ts_tablet_manager.cc:1398] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 778c78105092428bbcde75ab2cdb6a05: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20240501 14:10:35.778271 21530 raft_consensus.cc:1203] T 55830424a98348e382759c54afdeabcd P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:10:35.778967 19527 tablet_copy_service.cc:342] P 99a0be6ef2144b7fb247ba9d3f051d66: Request end of tablet copy session 778c78105092428bbcde75ab2cdb6a05-272eee3450cf4f48a4e1de2dcb28c0f7 received from {username='slave'} at 127.7.39.136:39441
I20240501 14:10:35.779057 19527 tablet_copy_service.cc:433] P 99a0be6ef2144b7fb247ba9d3f051d66: ending tablet copy session 778c78105092428bbcde75ab2cdb6a05-272eee3450cf4f48a4e1de2dcb28c0f7 on tablet 272eee3450cf4f48a4e1de2dcb28c0f7 with peer 778c78105092428bbcde75ab2cdb6a05
I20240501 14:10:35.781973 19636 raft_consensus.cc:2932] T ec2a46f8cfb3406082970e5fdb59b101 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:35.789988 19769 consensus_queue.cc:237] T bb291a0c743c4e638f79da36cc77faa3 P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
W20240501 14:10:35.791603 19702 consensus_peers.cc:485] T bb291a0c743c4e638f79da36cc77faa3 P da3c4e948ff747678f18456c134d7ed4 -> Peer 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136:34895): Couldn't send request to peer 778c78105092428bbcde75ab2cdb6a05. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bb291a0c743c4e638f79da36cc77faa3. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:35.792028 19376 raft_consensus.cc:1261] T bb291a0c743c4e638f79da36cc77faa3 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:35.792404 22411 consensus_queue.cc:1035] T bb291a0c743c4e638f79da36cc77faa3 P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:35.792536 19636 raft_consensus.cc:1261] T bb291a0c743c4e638f79da36cc77faa3 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:35.793025 22411 consensus_queue.cc:1035] T bb291a0c743c4e638f79da36cc77faa3 P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:35.805565 22878 raft_consensus.cc:2932] T bb291a0c743c4e638f79da36cc77faa3 P da3c4e948ff747678f18456c134d7ed4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:35.805922 19636 raft_consensus.cc:2932] T bb291a0c743c4e638f79da36cc77faa3 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:35.812968 19376 raft_consensus.cc:2932] T bb291a0c743c4e638f79da36cc77faa3 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:35.815924 19232 catalog_manager.cc:5484] T bb291a0c743c4e638f79da36cc77faa3 P fb1e6bc2d07b4ce3b622a66f14c93bdc reported cstate change: config changed from index -1 to 4, VOTER 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136) added. New cstate: current_term: 1 leader_uuid: "da3c4e948ff747678f18456c134d7ed4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:35.831363 23341 tablet_bootstrap.cc:492] T 410a7dd0dd734684a82875912966657d P 778c78105092428bbcde75ab2cdb6a05: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:35.831699 23341 tablet_bootstrap.cc:492] T 410a7dd0dd734684a82875912966657d P 778c78105092428bbcde75ab2cdb6a05: Bootstrap complete.
I20240501 14:10:35.831905 23341 ts_tablet_manager.cc:1367] T 410a7dd0dd734684a82875912966657d P 778c78105092428bbcde75ab2cdb6a05: Time spent bootstrapping tablet: real 0.095s user 0.025s sys 0.002s
I20240501 14:10:35.832417 23341 raft_consensus.cc:348] T 410a7dd0dd734684a82875912966657d P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:10:35.832594 23341 raft_consensus.cc:729] T 410a7dd0dd734684a82875912966657d P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 778c78105092428bbcde75ab2cdb6a05, State: Initialized, Role: FOLLOWER
I20240501 14:10:35.832716 23341 consensus_queue.cc:260] T 410a7dd0dd734684a82875912966657d P 778c78105092428bbcde75ab2cdb6a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:10:35.833593 23341 ts_tablet_manager.cc:1398] T 410a7dd0dd734684a82875912966657d P 778c78105092428bbcde75ab2cdb6a05: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:10:35.834071 19396 tablet_copy_service.cc:342] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Request end of tablet copy session 778c78105092428bbcde75ab2cdb6a05-410a7dd0dd734684a82875912966657d received from {username='slave'} at 127.7.39.136:56905
I20240501 14:10:35.834177 19396 tablet_copy_service.cc:433] P fb1e6bc2d07b4ce3b622a66f14c93bdc: ending tablet copy session 778c78105092428bbcde75ab2cdb6a05-410a7dd0dd734684a82875912966657d on tablet 410a7dd0dd734684a82875912966657d with peer 778c78105092428bbcde75ab2cdb6a05
I20240501 14:10:35.850927 19505 consensus_queue.cc:237] T f6ca088ce3aa4b97aa54c8ce6d700921 P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:35.852792 19376 raft_consensus.cc:1261] T f6ca088ce3aa4b97aa54c8ce6d700921 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Refusing update from remote peer 99a0be6ef2144b7fb247ba9d3f051d66: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:35.852861 19769 raft_consensus.cc:1261] T f6ca088ce3aa4b97aa54c8ce6d700921 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Refusing update from remote peer 99a0be6ef2144b7fb247ba9d3f051d66: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:35.853365 23095 consensus_queue.cc:1035] T f6ca088ce3aa4b97aa54c8ce6d700921 P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:35.853585 22081 consensus_queue.cc:1035] T f6ca088ce3aa4b97aa54c8ce6d700921 P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:10:35.854059 19440 consensus_peers.cc:485] T f6ca088ce3aa4b97aa54c8ce6d700921 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135:38751): Couldn't send request to peer 43ce4a4c954f4a5f80954b38933c189d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f6ca088ce3aa4b97aa54c8ce6d700921. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:35.856772 23095 raft_consensus.cc:2932] T f6ca088ce3aa4b97aa54c8ce6d700921 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:35.859515 19769 raft_consensus.cc:2932] T f6ca088ce3aa4b97aa54c8ce6d700921 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:35.861658 19376 raft_consensus.cc:2932] T f6ca088ce3aa4b97aa54c8ce6d700921 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:35.862282 19227 catalog_manager.cc:5484] T f6ca088ce3aa4b97aa54c8ce6d700921 P da3c4e948ff747678f18456c134d7ed4 reported cstate change: config changed from index -1 to 4, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New cstate: current_term: 1 leader_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:35.894630 21530 raft_consensus.cc:1203] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:10:35.902899 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:35.909127 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:35.917449 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:35.918720 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:35.922881 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:35.924343 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:35.924357 19769 consensus_queue.cc:237] T bb291a0c743c4e638f79da36cc77faa3 P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
W20240501 14:10:35.926703 19705 consensus_peers.cc:485] T bb291a0c743c4e638f79da36cc77faa3 P da3c4e948ff747678f18456c134d7ed4 -> Peer 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133:46883): Couldn't send request to peer 5bf81f3d986e428190d3f7727f36d96c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bb291a0c743c4e638f79da36cc77faa3. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:35.931756 19376 raft_consensus.cc:1261] T bb291a0c743c4e638f79da36cc77faa3 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:35.935448 22411 consensus_queue.cc:1035] T bb291a0c743c4e638f79da36cc77faa3 P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 14:10:35.936208 19702 consensus_peers.cc:485] T bb291a0c743c4e638f79da36cc77faa3 P da3c4e948ff747678f18456c134d7ed4 -> Peer 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136:34895): Couldn't send request to peer 778c78105092428bbcde75ab2cdb6a05. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bb291a0c743c4e638f79da36cc77faa3. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:35.938095 19636 raft_consensus.cc:1261] T bb291a0c743c4e638f79da36cc77faa3 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:35.938524 22411 consensus_queue.cc:1035] T bb291a0c743c4e638f79da36cc77faa3 P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:35.940738 21398 raft_consensus.cc:1203] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:10:35.941849 22877 raft_consensus.cc:2932] T bb291a0c743c4e638f79da36cc77faa3 P da3c4e948ff747678f18456c134d7ed4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:35.942255 19636 raft_consensus.cc:2932] T bb291a0c743c4e638f79da36cc77faa3 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:35.944849 19376 raft_consensus.cc:2932] T bb291a0c743c4e638f79da36cc77faa3 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:35.945197 19234 catalog_manager.cc:5484] T bb291a0c743c4e638f79da36cc77faa3 P ec8cde17a00645e7b38d5d29f7425367 reported cstate change: config changed from index 4 to 5, VOTER 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133) added. New cstate: current_term: 1 leader_uuid: "da3c4e948ff747678f18456c134d7ed4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:35.959264 21248 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:35.971508 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:36.038440 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:36.049230 21137 raft_consensus.cc:1203] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:10:36.056777 22800 ts_tablet_manager.cc:902] T 1d408b075d784953a57554bcdf19158a P 5bf81f3d986e428190d3f7727f36d96c: Initiating tablet copy from peer da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357)
I20240501 14:10:36.057225 22800 tablet_copy_client.cc:308] T 1d408b075d784953a57554bcdf19158a P 5bf81f3d986e428190d3f7727f36d96c: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.132:35357
I20240501 14:10:36.057683 19789 tablet_copy_service.cc:140] P da3c4e948ff747678f18456c134d7ed4: Received BeginTabletCopySession request for tablet 1d408b075d784953a57554bcdf19158a from peer 5bf81f3d986e428190d3f7727f36d96c ({username='slave'} at 127.7.39.133:51567)
I20240501 14:10:36.057799 19789 tablet_copy_service.cc:161] P da3c4e948ff747678f18456c134d7ed4: Beginning new tablet copy session on tablet 1d408b075d784953a57554bcdf19158a from peer 5bf81f3d986e428190d3f7727f36d96c at {username='slave'} at 127.7.39.133:51567: session id = 5bf81f3d986e428190d3f7727f36d96c-1d408b075d784953a57554bcdf19158a
I20240501 14:10:36.058756 19789 tablet_copy_source_session.cc:205] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:36.059155 23546 ts_tablet_manager.cc:902] T 2dcca6e0e6374520bd4522c80c110e97 P 43ce4a4c954f4a5f80954b38933c189d: Initiating tablet copy from peer ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201)
I20240501 14:10:36.059556 22800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d408b075d784953a57554bcdf19158a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:36.059599 23546 tablet_copy_client.cc:308] T 2dcca6e0e6374520bd4522c80c110e97 P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.131:45201
I20240501 14:10:36.060032 19657 tablet_copy_service.cc:140] P ec8cde17a00645e7b38d5d29f7425367: Received BeginTabletCopySession request for tablet 2dcca6e0e6374520bd4522c80c110e97 from peer 43ce4a4c954f4a5f80954b38933c189d ({username='slave'} at 127.7.39.135:35759)
I20240501 14:10:36.060130 19657 tablet_copy_service.cc:161] P ec8cde17a00645e7b38d5d29f7425367: Beginning new tablet copy session on tablet 2dcca6e0e6374520bd4522c80c110e97 from peer 43ce4a4c954f4a5f80954b38933c189d at {username='slave'} at 127.7.39.135:35759: session id = 43ce4a4c954f4a5f80954b38933c189d-2dcca6e0e6374520bd4522c80c110e97
I20240501 14:10:36.060786 19657 tablet_copy_source_session.cc:205] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:36.061519 22800 tablet_copy_client.cc:783] T 1d408b075d784953a57554bcdf19158a P 5bf81f3d986e428190d3f7727f36d96c: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:36.061533 23546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dcca6e0e6374520bd4522c80c110e97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:36.061695 22800 tablet_copy_client.cc:647] T 1d408b075d784953a57554bcdf19158a P 5bf81f3d986e428190d3f7727f36d96c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:36.062115 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:36.064913 23546 tablet_copy_client.cc:783] T 2dcca6e0e6374520bd4522c80c110e97 P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:36.065129 23546 tablet_copy_client.cc:647] T 2dcca6e0e6374520bd4522c80c110e97 P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:36.065399 22800 tablet_copy_client.cc:523] T 1d408b075d784953a57554bcdf19158a P 5bf81f3d986e428190d3f7727f36d96c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:36.066320 23546 tablet_copy_client.cc:523] T 2dcca6e0e6374520bd4522c80c110e97 P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:36.066607 22800 tablet_bootstrap.cc:492] T 1d408b075d784953a57554bcdf19158a P 5bf81f3d986e428190d3f7727f36d96c: Bootstrap starting.
I20240501 14:10:36.067927 23546 tablet_bootstrap.cc:492] T 2dcca6e0e6374520bd4522c80c110e97 P 43ce4a4c954f4a5f80954b38933c189d: Bootstrap starting.
I20240501 14:10:36.071712 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:36.076017 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:36.081357 21530 raft_consensus.cc:1203] T 410a7dd0dd734684a82875912966657d P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:10:36.088550 23558 ts_tablet_manager.cc:902] T 2dcca6e0e6374520bd4522c80c110e97 P e6246b183293412494919f1e7efc9448: Initiating tablet copy from peer ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201)
I20240501 14:10:36.090399 23558 tablet_copy_client.cc:308] T 2dcca6e0e6374520bd4522c80c110e97 P e6246b183293412494919f1e7efc9448: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.131:45201
I20240501 14:10:36.091003 19657 tablet_copy_service.cc:140] P ec8cde17a00645e7b38d5d29f7425367: Received BeginTabletCopySession request for tablet 2dcca6e0e6374520bd4522c80c110e97 from peer e6246b183293412494919f1e7efc9448 ({username='slave'} at 127.7.39.134:41913)
I20240501 14:10:36.091132 19657 tablet_copy_service.cc:161] P ec8cde17a00645e7b38d5d29f7425367: Beginning new tablet copy session on tablet 2dcca6e0e6374520bd4522c80c110e97 from peer e6246b183293412494919f1e7efc9448 at {username='slave'} at 127.7.39.134:41913: session id = e6246b183293412494919f1e7efc9448-2dcca6e0e6374520bd4522c80c110e97
I20240501 14:10:36.092072 19657 tablet_copy_source_session.cc:205] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:36.098942 23558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dcca6e0e6374520bd4522c80c110e97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:36.102041 23558 tablet_copy_client.cc:783] T 2dcca6e0e6374520bd4522c80c110e97 P e6246b183293412494919f1e7efc9448: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:36.103035 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:36.111423 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:36.111938 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:36.120798 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:36.124794 22800 tablet_bootstrap.cc:492] T 1d408b075d784953a57554bcdf19158a P 5bf81f3d986e428190d3f7727f36d96c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:36.125097 22800 tablet_bootstrap.cc:492] T 1d408b075d784953a57554bcdf19158a P 5bf81f3d986e428190d3f7727f36d96c: Bootstrap complete.
I20240501 14:10:36.125253 22800 ts_tablet_manager.cc:1367] T 1d408b075d784953a57554bcdf19158a P 5bf81f3d986e428190d3f7727f36d96c: Time spent bootstrapping tablet: real 0.059s user 0.022s sys 0.003s
I20240501 14:10:36.125733 22800 raft_consensus.cc:348] T 1d408b075d784953a57554bcdf19158a P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:36.125876 22800 raft_consensus.cc:729] T 1d408b075d784953a57554bcdf19158a P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf81f3d986e428190d3f7727f36d96c, State: Initialized, Role: FOLLOWER
I20240501 14:10:36.125994 22800 consensus_queue.cc:260] T 1d408b075d784953a57554bcdf19158a P 5bf81f3d986e428190d3f7727f36d96c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:36.126245 22800 ts_tablet_manager.cc:1398] T 1d408b075d784953a57554bcdf19158a P 5bf81f3d986e428190d3f7727f36d96c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:36.127123 19789 tablet_copy_service.cc:342] P da3c4e948ff747678f18456c134d7ed4: Request end of tablet copy session 5bf81f3d986e428190d3f7727f36d96c-1d408b075d784953a57554bcdf19158a received from {username='slave'} at 127.7.39.133:51567
I20240501 14:10:36.127218 19789 tablet_copy_service.cc:433] P da3c4e948ff747678f18456c134d7ed4: ending tablet copy session 5bf81f3d986e428190d3f7727f36d96c-1d408b075d784953a57554bcdf19158a on tablet 1d408b075d784953a57554bcdf19158a with peer 5bf81f3d986e428190d3f7727f36d96c
I20240501 14:10:36.133402 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:36.135825 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:36.103682 21248 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:36.141489 21247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:36.144567 23558 tablet_copy_client.cc:647] T 2dcca6e0e6374520bd4522c80c110e97 P e6246b183293412494919f1e7efc9448: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:36.145320 23579 ts_tablet_manager.cc:902] T fe554c3145de46cf8070efcecffbaedb P e6246b183293412494919f1e7efc9448: Initiating tablet copy from peer ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201)
I20240501 14:10:36.145857 23579 tablet_copy_client.cc:308] T fe554c3145de46cf8070efcecffbaedb P e6246b183293412494919f1e7efc9448: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.131:45201
I20240501 14:10:36.146350 19657 tablet_copy_service.cc:140] P ec8cde17a00645e7b38d5d29f7425367: Received BeginTabletCopySession request for tablet fe554c3145de46cf8070efcecffbaedb from peer e6246b183293412494919f1e7efc9448 ({username='slave'} at 127.7.39.134:41913)
I20240501 14:10:36.146452 19657 tablet_copy_service.cc:161] P ec8cde17a00645e7b38d5d29f7425367: Beginning new tablet copy session on tablet fe554c3145de46cf8070efcecffbaedb from peer e6246b183293412494919f1e7efc9448 at {username='slave'} at 127.7.39.134:41913: session id = e6246b183293412494919f1e7efc9448-fe554c3145de46cf8070efcecffbaedb
I20240501 14:10:36.147498 19657 tablet_copy_source_session.cc:205] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:36.152158 23558 tablet_copy_client.cc:523] T 2dcca6e0e6374520bd4522c80c110e97 P e6246b183293412494919f1e7efc9448: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:36.153713 23558 tablet_bootstrap.cc:492] T 2dcca6e0e6374520bd4522c80c110e97 P e6246b183293412494919f1e7efc9448: Bootstrap starting.
I20240501 14:10:36.156261 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:36.177754 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:36.184159 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:36.184492 23579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe554c3145de46cf8070efcecffbaedb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:36.187173 23579 tablet_copy_client.cc:783] T fe554c3145de46cf8070efcecffbaedb P e6246b183293412494919f1e7efc9448: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:36.188359 19637 consensus_queue.cc:237] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
W20240501 14:10:36.190593 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 661aaf83e3474966a257fcc039ef4784. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:36.190737 19505 raft_consensus.cc:1261] T 661aaf83e3474966a257fcc039ef4784 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:36.191699 23304 consensus_queue.cc:1035] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:36.193868 19769 raft_consensus.cc:1261] T 661aaf83e3474966a257fcc039ef4784 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:36.194379 23304 consensus_queue.cc:1035] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:36.214931 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:36.236451 23588 ts_tablet_manager.cc:902] T cc8e888d46d34a22b6d4409ec2ab53a9 P 43ce4a4c954f4a5f80954b38933c189d: Initiating tablet copy from peer ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201)
I20240501 14:10:36.237058 23588 tablet_copy_client.cc:308] T cc8e888d46d34a22b6d4409ec2ab53a9 P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.131:45201
I20240501 14:10:36.237295 23579 tablet_copy_client.cc:647] T fe554c3145de46cf8070efcecffbaedb P e6246b183293412494919f1e7efc9448: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:36.237579 19657 tablet_copy_service.cc:140] P ec8cde17a00645e7b38d5d29f7425367: Received BeginTabletCopySession request for tablet cc8e888d46d34a22b6d4409ec2ab53a9 from peer 43ce4a4c954f4a5f80954b38933c189d ({username='slave'} at 127.7.39.135:35759)
I20240501 14:10:36.237687 19657 tablet_copy_service.cc:161] P ec8cde17a00645e7b38d5d29f7425367: Beginning new tablet copy session on tablet cc8e888d46d34a22b6d4409ec2ab53a9 from peer 43ce4a4c954f4a5f80954b38933c189d at {username='slave'} at 127.7.39.135:35759: session id = 43ce4a4c954f4a5f80954b38933c189d-cc8e888d46d34a22b6d4409ec2ab53a9
I20240501 14:10:36.238577 23579 tablet_copy_client.cc:523] T fe554c3145de46cf8070efcecffbaedb P e6246b183293412494919f1e7efc9448: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:36.238590 19657 tablet_copy_source_session.cc:205] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:36.239683 23588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc8e888d46d34a22b6d4409ec2ab53a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:36.240120 23579 tablet_bootstrap.cc:492] T fe554c3145de46cf8070efcecffbaedb P e6246b183293412494919f1e7efc9448: Bootstrap starting.
I20240501 14:10:36.248485 21530 raft_consensus.cc:1203] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:10:36.248754 23596 ts_tablet_manager.cc:902] T d63afd081986473aa2715d074b123a93 P e6246b183293412494919f1e7efc9448: Initiating tablet copy from peer fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847)
I20240501 14:10:36.249217 23596 tablet_copy_client.cc:308] T d63afd081986473aa2715d074b123a93 P e6246b183293412494919f1e7efc9448: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.129:35847
I20240501 14:10:36.251938 19396 tablet_copy_service.cc:140] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Received BeginTabletCopySession request for tablet d63afd081986473aa2715d074b123a93 from peer e6246b183293412494919f1e7efc9448 ({username='slave'} at 127.7.39.134:34039)
I20240501 14:10:36.252051 19396 tablet_copy_service.cc:161] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Beginning new tablet copy session on tablet d63afd081986473aa2715d074b123a93 from peer e6246b183293412494919f1e7efc9448 at {username='slave'} at 127.7.39.134:34039: session id = e6246b183293412494919f1e7efc9448-d63afd081986473aa2715d074b123a93
I20240501 14:10:36.252991 19396 tablet_copy_source_session.cc:205] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:36.253782 22086 raft_consensus.cc:2932] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:36.254369 19769 raft_consensus.cc:2932] T 661aaf83e3474966a257fcc039ef4784 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:36.256564 19505 raft_consensus.cc:2932] T 661aaf83e3474966a257fcc039ef4784 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:36.257637 23632 ts_tablet_manager.cc:902] T cc8e888d46d34a22b6d4409ec2ab53a9 P e6246b183293412494919f1e7efc9448: Initiating tablet copy from peer ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201)
I20240501 14:10:36.258854 19227 catalog_manager.cc:5484] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 reported cstate change: config changed from index -1 to 4, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:36.259712 23596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d63afd081986473aa2715d074b123a93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:36.262188 23596 tablet_copy_client.cc:783] T d63afd081986473aa2715d074b123a93 P e6246b183293412494919f1e7efc9448: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:36.262428 23596 tablet_copy_client.cc:647] T d63afd081986473aa2715d074b123a93 P e6246b183293412494919f1e7efc9448: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:36.264003 23632 tablet_copy_client.cc:308] T cc8e888d46d34a22b6d4409ec2ab53a9 P e6246b183293412494919f1e7efc9448: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.131:45201
I20240501 14:10:36.264508 19657 tablet_copy_service.cc:140] P ec8cde17a00645e7b38d5d29f7425367: Received BeginTabletCopySession request for tablet cc8e888d46d34a22b6d4409ec2ab53a9 from peer e6246b183293412494919f1e7efc9448 ({username='slave'} at 127.7.39.134:41913)
I20240501 14:10:36.264608 19657 tablet_copy_service.cc:161] P ec8cde17a00645e7b38d5d29f7425367: Beginning new tablet copy session on tablet cc8e888d46d34a22b6d4409ec2ab53a9 from peer e6246b183293412494919f1e7efc9448 at {username='slave'} at 127.7.39.134:41913: session id = e6246b183293412494919f1e7efc9448-cc8e888d46d34a22b6d4409ec2ab53a9
I20240501 14:10:36.265439 19657 tablet_copy_source_session.cc:205] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:36.275147 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:36.275954 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:36.285482 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:36.294459 23632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc8e888d46d34a22b6d4409ec2ab53a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:36.296941 23579 tablet_bootstrap.cc:492] T fe554c3145de46cf8070efcecffbaedb P e6246b183293412494919f1e7efc9448: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:36.297315 23579 tablet_bootstrap.cc:492] T fe554c3145de46cf8070efcecffbaedb P e6246b183293412494919f1e7efc9448: Bootstrap complete.
I20240501 14:10:36.297544 23579 ts_tablet_manager.cc:1367] T fe554c3145de46cf8070efcecffbaedb P e6246b183293412494919f1e7efc9448: Time spent bootstrapping tablet: real 0.057s user 0.033s sys 0.000s
I20240501 14:10:36.298017 23579 raft_consensus.cc:348] T fe554c3145de46cf8070efcecffbaedb P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:36.298213 23579 raft_consensus.cc:729] T fe554c3145de46cf8070efcecffbaedb P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6246b183293412494919f1e7efc9448, State: Initialized, Role: FOLLOWER
I20240501 14:10:36.298409 23579 consensus_queue.cc:260] T fe554c3145de46cf8070efcecffbaedb P e6246b183293412494919f1e7efc9448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:36.298892 23579 ts_tablet_manager.cc:1398] T fe554c3145de46cf8070efcecffbaedb P e6246b183293412494919f1e7efc9448: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:36.299465 19657 tablet_copy_service.cc:342] P ec8cde17a00645e7b38d5d29f7425367: Request end of tablet copy session e6246b183293412494919f1e7efc9448-fe554c3145de46cf8070efcecffbaedb received from {username='slave'} at 127.7.39.134:41913
I20240501 14:10:36.299552 19657 tablet_copy_service.cc:433] P ec8cde17a00645e7b38d5d29f7425367: ending tablet copy session e6246b183293412494919f1e7efc9448-fe554c3145de46cf8070efcecffbaedb on tablet fe554c3145de46cf8070efcecffbaedb with peer e6246b183293412494919f1e7efc9448
I20240501 14:10:36.299708 23596 tablet_copy_client.cc:523] T d63afd081986473aa2715d074b123a93 P e6246b183293412494919f1e7efc9448: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:36.301234 23596 tablet_bootstrap.cc:492] T d63afd081986473aa2715d074b123a93 P e6246b183293412494919f1e7efc9448: Bootstrap starting.
I20240501 14:10:36.302532 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:36.303640 23341 ts_tablet_manager.cc:902] T ec2a46f8cfb3406082970e5fdb59b101 P 778c78105092428bbcde75ab2cdb6a05: Initiating tablet copy from peer da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357)
I20240501 14:10:36.304064 23341 tablet_copy_client.cc:308] T ec2a46f8cfb3406082970e5fdb59b101 P 778c78105092428bbcde75ab2cdb6a05: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.132:35357
I20240501 14:10:36.304476 19789 tablet_copy_service.cc:140] P da3c4e948ff747678f18456c134d7ed4: Received BeginTabletCopySession request for tablet ec2a46f8cfb3406082970e5fdb59b101 from peer 778c78105092428bbcde75ab2cdb6a05 ({username='slave'} at 127.7.39.136:40943)
I20240501 14:10:36.304582 19789 tablet_copy_service.cc:161] P da3c4e948ff747678f18456c134d7ed4: Beginning new tablet copy session on tablet ec2a46f8cfb3406082970e5fdb59b101 from peer 778c78105092428bbcde75ab2cdb6a05 at {username='slave'} at 127.7.39.136:40943: session id = 778c78105092428bbcde75ab2cdb6a05-ec2a46f8cfb3406082970e5fdb59b101
I20240501 14:10:36.305469 19789 tablet_copy_source_session.cc:205] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:36.312958 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:36.313256 21247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:36.314316 23341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec2a46f8cfb3406082970e5fdb59b101. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:36.316510 23341 tablet_copy_client.cc:783] T ec2a46f8cfb3406082970e5fdb59b101 P 778c78105092428bbcde75ab2cdb6a05: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:36.316665 23341 tablet_copy_client.cc:647] T ec2a46f8cfb3406082970e5fdb59b101 P 778c78105092428bbcde75ab2cdb6a05: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:36.317255 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:36.319036 23588 tablet_copy_client.cc:783] T cc8e888d46d34a22b6d4409ec2ab53a9 P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:36.319293 23588 tablet_copy_client.cc:647] T cc8e888d46d34a22b6d4409ec2ab53a9 P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:36.320371 22800 ts_tablet_manager.cc:902] T bb291a0c743c4e638f79da36cc77faa3 P 5bf81f3d986e428190d3f7727f36d96c: Initiating tablet copy from peer da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357)
I20240501 14:10:36.320827 22800 tablet_copy_client.cc:308] T bb291a0c743c4e638f79da36cc77faa3 P 5bf81f3d986e428190d3f7727f36d96c: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.132:35357
I20240501 14:10:36.321316 19789 tablet_copy_service.cc:140] P da3c4e948ff747678f18456c134d7ed4: Received BeginTabletCopySession request for tablet bb291a0c743c4e638f79da36cc77faa3 from peer 5bf81f3d986e428190d3f7727f36d96c ({username='slave'} at 127.7.39.133:51567)
I20240501 14:10:36.321424 19789 tablet_copy_service.cc:161] P da3c4e948ff747678f18456c134d7ed4: Beginning new tablet copy session on tablet bb291a0c743c4e638f79da36cc77faa3 from peer 5bf81f3d986e428190d3f7727f36d96c at {username='slave'} at 127.7.39.133:51567: session id = 5bf81f3d986e428190d3f7727f36d96c-bb291a0c743c4e638f79da36cc77faa3
I20240501 14:10:36.322341 19789 tablet_copy_source_session.cc:205] T bb291a0c743c4e638f79da36cc77faa3 P da3c4e948ff747678f18456c134d7ed4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:36.323786 22800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb291a0c743c4e638f79da36cc77faa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:36.325843 22800 tablet_copy_client.cc:783] T bb291a0c743c4e638f79da36cc77faa3 P 5bf81f3d986e428190d3f7727f36d96c: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:36.326283 22800 tablet_copy_client.cc:647] T bb291a0c743c4e638f79da36cc77faa3 P 5bf81f3d986e428190d3f7727f36d96c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:36.339383 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:36.348462 23632 tablet_copy_client.cc:783] T cc8e888d46d34a22b6d4409ec2ab53a9 P e6246b183293412494919f1e7efc9448: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:36.348774 23632 tablet_copy_client.cc:647] T cc8e888d46d34a22b6d4409ec2ab53a9 P e6246b183293412494919f1e7efc9448: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:36.350855 23588 tablet_copy_client.cc:523] T cc8e888d46d34a22b6d4409ec2ab53a9 P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:36.352596 23588 tablet_bootstrap.cc:492] T cc8e888d46d34a22b6d4409ec2ab53a9 P 43ce4a4c954f4a5f80954b38933c189d: Bootstrap starting.
I20240501 14:10:36.363497 22800 tablet_copy_client.cc:523] T bb291a0c743c4e638f79da36cc77faa3 P 5bf81f3d986e428190d3f7727f36d96c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:36.364893 22800 tablet_bootstrap.cc:492] T bb291a0c743c4e638f79da36cc77faa3 P 5bf81f3d986e428190d3f7727f36d96c: Bootstrap starting.
I20240501 14:10:36.367091 23632 tablet_copy_client.cc:523] T cc8e888d46d34a22b6d4409ec2ab53a9 P e6246b183293412494919f1e7efc9448: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:36.368996 23632 tablet_bootstrap.cc:492] T cc8e888d46d34a22b6d4409ec2ab53a9 P e6246b183293412494919f1e7efc9448: Bootstrap starting.
I20240501 14:10:36.375265 23341 tablet_copy_client.cc:523] T ec2a46f8cfb3406082970e5fdb59b101 P 778c78105092428bbcde75ab2cdb6a05: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:36.376645 23341 tablet_bootstrap.cc:492] T ec2a46f8cfb3406082970e5fdb59b101 P 778c78105092428bbcde75ab2cdb6a05: Bootstrap starting.
I20240501 14:10:36.378985 23596 tablet_bootstrap.cc:492] T d63afd081986473aa2715d074b123a93 P e6246b183293412494919f1e7efc9448: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:36.379279 23596 tablet_bootstrap.cc:492] T d63afd081986473aa2715d074b123a93 P e6246b183293412494919f1e7efc9448: Bootstrap complete.
I20240501 14:10:36.379504 23596 ts_tablet_manager.cc:1367] T d63afd081986473aa2715d074b123a93 P e6246b183293412494919f1e7efc9448: Time spent bootstrapping tablet: real 0.078s user 0.028s sys 0.000s
I20240501 14:10:36.380025 23596 raft_consensus.cc:348] T d63afd081986473aa2715d074b123a93 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:36.380195 23596 raft_consensus.cc:729] T d63afd081986473aa2715d074b123a93 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6246b183293412494919f1e7efc9448, State: Initialized, Role: FOLLOWER
I20240501 14:10:36.380379 23596 consensus_queue.cc:260] T d63afd081986473aa2715d074b123a93 P e6246b183293412494919f1e7efc9448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:36.380703 23596 ts_tablet_manager.cc:1398] T d63afd081986473aa2715d074b123a93 P e6246b183293412494919f1e7efc9448: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:36.381343 19396 tablet_copy_service.cc:342] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Request end of tablet copy session e6246b183293412494919f1e7efc9448-d63afd081986473aa2715d074b123a93 received from {username='slave'} at 127.7.39.134:34039
I20240501 14:10:36.381428 19396 tablet_copy_service.cc:433] P fb1e6bc2d07b4ce3b622a66f14c93bdc: ending tablet copy session e6246b183293412494919f1e7efc9448-d63afd081986473aa2715d074b123a93 on tablet d63afd081986473aa2715d074b123a93 with peer e6246b183293412494919f1e7efc9448
I20240501 14:10:36.401088 23546 tablet_bootstrap.cc:492] T 2dcca6e0e6374520bd4522c80c110e97 P 43ce4a4c954f4a5f80954b38933c189d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:36.401568 23546 tablet_bootstrap.cc:492] T 2dcca6e0e6374520bd4522c80c110e97 P 43ce4a4c954f4a5f80954b38933c189d: Bootstrap complete.
I20240501 14:10:36.401829 23546 ts_tablet_manager.cc:1367] T 2dcca6e0e6374520bd4522c80c110e97 P 43ce4a4c954f4a5f80954b38933c189d: Time spent bootstrapping tablet: real 0.334s user 0.027s sys 0.001s
I20240501 14:10:36.403759 23546 raft_consensus.cc:348] T 2dcca6e0e6374520bd4522c80c110e97 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:36.403884 23546 raft_consensus.cc:729] T 2dcca6e0e6374520bd4522c80c110e97 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43ce4a4c954f4a5f80954b38933c189d, State: Initialized, Role: FOLLOWER
I20240501 14:10:36.408758 23680 ts_tablet_manager.cc:902] T f6ca088ce3aa4b97aa54c8ce6d700921 P 43ce4a4c954f4a5f80954b38933c189d: Initiating tablet copy from peer 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829)
I20240501 14:10:36.409466 23680 tablet_copy_client.cc:308] T f6ca088ce3aa4b97aa54c8ce6d700921 P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.130:41829
I20240501 14:10:36.410375 19527 tablet_copy_service.cc:140] P 99a0be6ef2144b7fb247ba9d3f051d66: Received BeginTabletCopySession request for tablet f6ca088ce3aa4b97aa54c8ce6d700921 from peer 43ce4a4c954f4a5f80954b38933c189d ({username='slave'} at 127.7.39.135:44769)
I20240501 14:10:36.410481 19527 tablet_copy_service.cc:161] P 99a0be6ef2144b7fb247ba9d3f051d66: Beginning new tablet copy session on tablet f6ca088ce3aa4b97aa54c8ce6d700921 from peer 43ce4a4c954f4a5f80954b38933c189d at {username='slave'} at 127.7.39.135:44769: session id = 43ce4a4c954f4a5f80954b38933c189d-f6ca088ce3aa4b97aa54c8ce6d700921
I20240501 14:10:36.411429 19527 tablet_copy_source_session.cc:205] T f6ca088ce3aa4b97aa54c8ce6d700921 P 99a0be6ef2144b7fb247ba9d3f051d66: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:36.413816 23680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6ca088ce3aa4b97aa54c8ce6d700921. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:36.416149 23680 tablet_copy_client.cc:783] T f6ca088ce3aa4b97aa54c8ce6d700921 P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:36.416388 23680 tablet_copy_client.cc:647] T f6ca088ce3aa4b97aa54c8ce6d700921 P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:36.417869 23680 tablet_copy_client.cc:523] T f6ca088ce3aa4b97aa54c8ce6d700921 P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:36.419222 23680 tablet_bootstrap.cc:492] T f6ca088ce3aa4b97aa54c8ce6d700921 P 43ce4a4c954f4a5f80954b38933c189d: Bootstrap starting.
I20240501 14:10:36.427301 23341 tablet_bootstrap.cc:492] T ec2a46f8cfb3406082970e5fdb59b101 P 778c78105092428bbcde75ab2cdb6a05: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:36.428005 23341 tablet_bootstrap.cc:492] T ec2a46f8cfb3406082970e5fdb59b101 P 778c78105092428bbcde75ab2cdb6a05: Bootstrap complete.
I20240501 14:10:36.428243 23588 tablet_bootstrap.cc:492] T cc8e888d46d34a22b6d4409ec2ab53a9 P 43ce4a4c954f4a5f80954b38933c189d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:36.428798 23341 ts_tablet_manager.cc:1367] T ec2a46f8cfb3406082970e5fdb59b101 P 778c78105092428bbcde75ab2cdb6a05: Time spent bootstrapping tablet: real 0.052s user 0.029s sys 0.000s
I20240501 14:10:36.429417 23341 raft_consensus.cc:348] T ec2a46f8cfb3406082970e5fdb59b101 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:10:36.429674 23341 raft_consensus.cc:729] T ec2a46f8cfb3406082970e5fdb59b101 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 778c78105092428bbcde75ab2cdb6a05, State: Initialized, Role: FOLLOWER
I20240501 14:10:36.429924 23341 consensus_queue.cc:260] T ec2a46f8cfb3406082970e5fdb59b101 P 778c78105092428bbcde75ab2cdb6a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:10:36.430298 23341 ts_tablet_manager.cc:1398] T ec2a46f8cfb3406082970e5fdb59b101 P 778c78105092428bbcde75ab2cdb6a05: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:36.431149 19789 tablet_copy_service.cc:342] P da3c4e948ff747678f18456c134d7ed4: Request end of tablet copy session 778c78105092428bbcde75ab2cdb6a05-ec2a46f8cfb3406082970e5fdb59b101 received from {username='slave'} at 127.7.39.136:40943
I20240501 14:10:36.431224 19789 tablet_copy_service.cc:433] P da3c4e948ff747678f18456c134d7ed4: ending tablet copy session 778c78105092428bbcde75ab2cdb6a05-ec2a46f8cfb3406082970e5fdb59b101 on tablet ec2a46f8cfb3406082970e5fdb59b101 with peer 778c78105092428bbcde75ab2cdb6a05
I20240501 14:10:36.435139 23546 consensus_queue.cc:260] T 2dcca6e0e6374520bd4522c80c110e97 P 43ce4a4c954f4a5f80954b38933c189d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:36.435582 23546 ts_tablet_manager.cc:1398] T 2dcca6e0e6374520bd4522c80c110e97 P 43ce4a4c954f4a5f80954b38933c189d: Time spent starting tablet: real 0.034s user 0.001s sys 0.000s
I20240501 14:10:36.436076 19657 tablet_copy_service.cc:342] P ec8cde17a00645e7b38d5d29f7425367: Request end of tablet copy session 43ce4a4c954f4a5f80954b38933c189d-2dcca6e0e6374520bd4522c80c110e97 received from {username='slave'} at 127.7.39.135:35759
I20240501 14:10:36.436159 19657 tablet_copy_service.cc:433] P ec8cde17a00645e7b38d5d29f7425367: ending tablet copy session 43ce4a4c954f4a5f80954b38933c189d-2dcca6e0e6374520bd4522c80c110e97 on tablet 2dcca6e0e6374520bd4522c80c110e97 with peer 43ce4a4c954f4a5f80954b38933c189d
I20240501 14:10:36.439294 23588 tablet_bootstrap.cc:492] T cc8e888d46d34a22b6d4409ec2ab53a9 P 43ce4a4c954f4a5f80954b38933c189d: Bootstrap complete.
I20240501 14:10:36.439482 23588 ts_tablet_manager.cc:1367] T cc8e888d46d34a22b6d4409ec2ab53a9 P 43ce4a4c954f4a5f80954b38933c189d: Time spent bootstrapping tablet: real 0.087s user 0.025s sys 0.002s
I20240501 14:10:36.439905 23588 raft_consensus.cc:348] T cc8e888d46d34a22b6d4409ec2ab53a9 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:36.440013 23588 raft_consensus.cc:729] T cc8e888d46d34a22b6d4409ec2ab53a9 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43ce4a4c954f4a5f80954b38933c189d, State: Initialized, Role: FOLLOWER
I20240501 14:10:36.440127 23588 consensus_queue.cc:260] T cc8e888d46d34a22b6d4409ec2ab53a9 P 43ce4a4c954f4a5f80954b38933c189d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:36.440385 23588 ts_tablet_manager.cc:1398] T cc8e888d46d34a22b6d4409ec2ab53a9 P 43ce4a4c954f4a5f80954b38933c189d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:36.441028 19657 tablet_copy_service.cc:342] P ec8cde17a00645e7b38d5d29f7425367: Request end of tablet copy session 43ce4a4c954f4a5f80954b38933c189d-cc8e888d46d34a22b6d4409ec2ab53a9 received from {username='slave'} at 127.7.39.135:35759
I20240501 14:10:36.441107 19657 tablet_copy_service.cc:433] P ec8cde17a00645e7b38d5d29f7425367: ending tablet copy session 43ce4a4c954f4a5f80954b38933c189d-cc8e888d46d34a22b6d4409ec2ab53a9 on tablet cc8e888d46d34a22b6d4409ec2ab53a9 with peer 43ce4a4c954f4a5f80954b38933c189d
I20240501 14:10:36.449128 19769 consensus_queue.cc:237] T dab5e5722e2e4b0eaea73288d8dce265 P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:36.450788 19637 raft_consensus.cc:1261] T dab5e5722e2e4b0eaea73288d8dce265 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:36.451519 19505 raft_consensus.cc:1261] T dab5e5722e2e4b0eaea73288d8dce265 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:36.452142 22411 consensus_queue.cc:1035] T dab5e5722e2e4b0eaea73288d8dce265 P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:36.454386 22411 consensus_queue.cc:1035] T dab5e5722e2e4b0eaea73288d8dce265 P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:10:36.455682 19705 consensus_peers.cc:485] T dab5e5722e2e4b0eaea73288d8dce265 P da3c4e948ff747678f18456c134d7ed4 -> Peer 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133:46883): Couldn't send request to peer 5bf81f3d986e428190d3f7727f36d96c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dab5e5722e2e4b0eaea73288d8dce265. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:36.455832 19636 consensus_queue.cc:237] T 12ffa74f7a5b472aac57253693273cad P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
W20240501 14:10:36.457433 19572 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 12ffa74f7a5b472aac57253693273cad. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:36.457527 19376 raft_consensus.cc:1261] T 12ffa74f7a5b472aac57253693273cad P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:36.457945 23303 consensus_queue.cc:1035] T 12ffa74f7a5b472aac57253693273cad P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:36.458880 19769 raft_consensus.cc:1261] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:36.459303 23303 consensus_queue.cc:1035] T 12ffa74f7a5b472aac57253693273cad P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:36.461159 23707 raft_consensus.cc:2932] T dab5e5722e2e4b0eaea73288d8dce265 P da3c4e948ff747678f18456c134d7ed4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:36.463604 19505 raft_consensus.cc:2932] T dab5e5722e2e4b0eaea73288d8dce265 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:36.465342 19636 raft_consensus.cc:2932] T dab5e5722e2e4b0eaea73288d8dce265 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:36.465801 23558 tablet_bootstrap.cc:492] T 2dcca6e0e6374520bd4522c80c110e97 P e6246b183293412494919f1e7efc9448: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:36.466112 23558 tablet_bootstrap.cc:492] T 2dcca6e0e6374520bd4522c80c110e97 P e6246b183293412494919f1e7efc9448: Bootstrap complete.
I20240501 14:10:36.466286 23558 ts_tablet_manager.cc:1367] T 2dcca6e0e6374520bd4522c80c110e97 P e6246b183293412494919f1e7efc9448: Time spent bootstrapping tablet: real 0.313s user 0.024s sys 0.005s
I20240501 14:10:36.466737 23558 raft_consensus.cc:348] T 2dcca6e0e6374520bd4522c80c110e97 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:36.466849 23558 raft_consensus.cc:729] T 2dcca6e0e6374520bd4522c80c110e97 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6246b183293412494919f1e7efc9448, State: Initialized, Role: FOLLOWER
I20240501 14:10:36.469178 19231 catalog_manager.cc:5484] T dab5e5722e2e4b0eaea73288d8dce265 P 99a0be6ef2144b7fb247ba9d3f051d66 reported cstate change: config changed from index -1 to 4, VOTER 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133) added. New cstate: current_term: 1 leader_uuid: "da3c4e948ff747678f18456c134d7ed4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:36.466979 23558 consensus_queue.cc:260] T 2dcca6e0e6374520bd4522c80c110e97 P e6246b183293412494919f1e7efc9448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:36.476893 23558 ts_tablet_manager.cc:1398] T 2dcca6e0e6374520bd4522c80c110e97 P e6246b183293412494919f1e7efc9448: Time spent starting tablet: real 0.011s user 0.004s sys 0.000s
I20240501 14:10:36.478075 19657 tablet_copy_service.cc:342] P ec8cde17a00645e7b38d5d29f7425367: Request end of tablet copy session e6246b183293412494919f1e7efc9448-2dcca6e0e6374520bd4522c80c110e97 received from {username='slave'} at 127.7.39.134:41913
I20240501 14:10:36.478159 19657 tablet_copy_service.cc:433] P ec8cde17a00645e7b38d5d29f7425367: ending tablet copy session e6246b183293412494919f1e7efc9448-2dcca6e0e6374520bd4522c80c110e97 on tablet 2dcca6e0e6374520bd4522c80c110e97 with peer e6246b183293412494919f1e7efc9448
I20240501 14:10:36.478714 23303 raft_consensus.cc:2932] T 12ffa74f7a5b472aac57253693273cad P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:36.479990 19769 raft_consensus.cc:2932] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:36.482452 19225 catalog_manager.cc:5484] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 reported cstate change: config changed from index -1 to 4, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:36.485391 19376 raft_consensus.cc:2932] T 12ffa74f7a5b472aac57253693273cad P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:36.492004 23680 tablet_bootstrap.cc:492] T f6ca088ce3aa4b97aa54c8ce6d700921 P 43ce4a4c954f4a5f80954b38933c189d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:36.492900 23680 tablet_bootstrap.cc:492] T f6ca088ce3aa4b97aa54c8ce6d700921 P 43ce4a4c954f4a5f80954b38933c189d: Bootstrap complete.
I20240501 14:10:36.493731 23680 ts_tablet_manager.cc:1367] T f6ca088ce3aa4b97aa54c8ce6d700921 P 43ce4a4c954f4a5f80954b38933c189d: Time spent bootstrapping tablet: real 0.075s user 0.024s sys 0.003s
I20240501 14:10:36.495815 23680 raft_consensus.cc:348] T f6ca088ce3aa4b97aa54c8ce6d700921 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:36.496026 23680 raft_consensus.cc:729] T f6ca088ce3aa4b97aa54c8ce6d700921 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43ce4a4c954f4a5f80954b38933c189d, State: Initialized, Role: FOLLOWER
I20240501 14:10:36.496799 23680 consensus_queue.cc:260] T f6ca088ce3aa4b97aa54c8ce6d700921 P 43ce4a4c954f4a5f80954b38933c189d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:36.497195 23680 ts_tablet_manager.cc:1398] T f6ca088ce3aa4b97aa54c8ce6d700921 P 43ce4a4c954f4a5f80954b38933c189d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:10:36.498095 19527 tablet_copy_service.cc:342] P 99a0be6ef2144b7fb247ba9d3f051d66: Request end of tablet copy session 43ce4a4c954f4a5f80954b38933c189d-f6ca088ce3aa4b97aa54c8ce6d700921 received from {username='slave'} at 127.7.39.135:44769
I20240501 14:10:36.498200 19527 tablet_copy_service.cc:433] P 99a0be6ef2144b7fb247ba9d3f051d66: ending tablet copy session 43ce4a4c954f4a5f80954b38933c189d-f6ca088ce3aa4b97aa54c8ce6d700921 on tablet f6ca088ce3aa4b97aa54c8ce6d700921 with peer 43ce4a4c954f4a5f80954b38933c189d
I20240501 14:10:36.499167 21398 raft_consensus.cc:1203] T 2dcca6e0e6374520bd4522c80c110e97 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:10:36.504137 22800 tablet_bootstrap.cc:492] T bb291a0c743c4e638f79da36cc77faa3 P 5bf81f3d986e428190d3f7727f36d96c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:36.504503 22800 tablet_bootstrap.cc:492] T bb291a0c743c4e638f79da36cc77faa3 P 5bf81f3d986e428190d3f7727f36d96c: Bootstrap complete.
I20240501 14:10:36.504739 22800 ts_tablet_manager.cc:1367] T bb291a0c743c4e638f79da36cc77faa3 P 5bf81f3d986e428190d3f7727f36d96c: Time spent bootstrapping tablet: real 0.140s user 0.027s sys 0.000s
I20240501 14:10:36.505198 22800 raft_consensus.cc:348] T bb291a0c743c4e638f79da36cc77faa3 P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:36.505303 22800 raft_consensus.cc:729] T bb291a0c743c4e638f79da36cc77faa3 P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf81f3d986e428190d3f7727f36d96c, State: Initialized, Role: FOLLOWER
I20240501 14:10:36.505467 22800 consensus_queue.cc:260] T bb291a0c743c4e638f79da36cc77faa3 P 5bf81f3d986e428190d3f7727f36d96c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:36.505765 22800 ts_tablet_manager.cc:1398] T bb291a0c743c4e638f79da36cc77faa3 P 5bf81f3d986e428190d3f7727f36d96c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:36.506260 19789 tablet_copy_service.cc:342] P da3c4e948ff747678f18456c134d7ed4: Request end of tablet copy session 5bf81f3d986e428190d3f7727f36d96c-bb291a0c743c4e638f79da36cc77faa3 received from {username='slave'} at 127.7.39.133:51567
I20240501 14:10:36.506348 19789 tablet_copy_service.cc:433] P da3c4e948ff747678f18456c134d7ed4: ending tablet copy session 5bf81f3d986e428190d3f7727f36d96c-bb291a0c743c4e638f79da36cc77faa3 on tablet bb291a0c743c4e638f79da36cc77faa3 with peer 5bf81f3d986e428190d3f7727f36d96c
I20240501 14:10:36.513020 23632 tablet_bootstrap.cc:492] T cc8e888d46d34a22b6d4409ec2ab53a9 P e6246b183293412494919f1e7efc9448: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:36.513362 23632 tablet_bootstrap.cc:492] T cc8e888d46d34a22b6d4409ec2ab53a9 P e6246b183293412494919f1e7efc9448: Bootstrap complete.
I20240501 14:10:36.514231 23632 ts_tablet_manager.cc:1367] T cc8e888d46d34a22b6d4409ec2ab53a9 P e6246b183293412494919f1e7efc9448: Time spent bootstrapping tablet: real 0.145s user 0.026s sys 0.003s
I20240501 14:10:36.514689 23632 raft_consensus.cc:348] T cc8e888d46d34a22b6d4409ec2ab53a9 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:36.514894 23632 raft_consensus.cc:729] T cc8e888d46d34a22b6d4409ec2ab53a9 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6246b183293412494919f1e7efc9448, State: Initialized, Role: FOLLOWER
I20240501 14:10:36.514869 19636 consensus_queue.cc:237] T dadab886b03540579d138cdb5b25e286 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:10:36.515197 23632 consensus_queue.cc:260] T cc8e888d46d34a22b6d4409ec2ab53a9 P e6246b183293412494919f1e7efc9448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:36.515623 23632 ts_tablet_manager.cc:1398] T cc8e888d46d34a22b6d4409ec2ab53a9 P e6246b183293412494919f1e7efc9448: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:36.516563 19505 raft_consensus.cc:1261] T dadab886b03540579d138cdb5b25e286 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:36.517329 19657 tablet_copy_service.cc:342] P ec8cde17a00645e7b38d5d29f7425367: Request end of tablet copy session e6246b183293412494919f1e7efc9448-cc8e888d46d34a22b6d4409ec2ab53a9 received from {username='slave'} at 127.7.39.134:41913
I20240501 14:10:36.517407 19657 tablet_copy_service.cc:433] P ec8cde17a00645e7b38d5d29f7425367: ending tablet copy session e6246b183293412494919f1e7efc9448-cc8e888d46d34a22b6d4409ec2ab53a9 on tablet cc8e888d46d34a22b6d4409ec2ab53a9 with peer e6246b183293412494919f1e7efc9448
I20240501 14:10:36.519297 23303 consensus_queue.cc:1035] T dadab886b03540579d138cdb5b25e286 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:36.519410 19376 raft_consensus.cc:1261] T dadab886b03540579d138cdb5b25e286 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:10:36.519845 19571 consensus_peers.cc:485] T dadab886b03540579d138cdb5b25e286 P ec8cde17a00645e7b38d5d29f7425367 -> Peer 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136:34895): Couldn't send request to peer 778c78105092428bbcde75ab2cdb6a05. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dadab886b03540579d138cdb5b25e286. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:36.520133 23303 consensus_queue.cc:1035] T dadab886b03540579d138cdb5b25e286 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:36.520944 19376 consensus_queue.cc:237] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:36.522372 23303 raft_consensus.cc:2932] T dadab886b03540579d138cdb5b25e286 P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:36.522780 19769 raft_consensus.cc:1261] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:36.523195 22527 consensus_queue.cc:1035] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:36.523423 19505 raft_consensus.cc:1261] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:10:36.523797 19312 consensus_peers.cc:485] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133:46883): Couldn't send request to peer 5bf81f3d986e428190d3f7727f36d96c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 78e14dea8b6e4c0abd1c88dcd5b4e608. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:36.524758 22527 consensus_queue.cc:1035] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:36.525000 19375 raft_consensus.cc:2932] T dadab886b03540579d138cdb5b25e286 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:36.525187 19505 raft_consensus.cc:2932] T dadab886b03540579d138cdb5b25e286 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:36.525451 19636 consensus_queue.cc:237] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:10:36.527787 19767 raft_consensus.cc:1261] T fe554c3145de46cf8070efcecffbaedb P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:36.527806 23341 ts_tablet_manager.cc:902] T bb291a0c743c4e638f79da36cc77faa3 P 778c78105092428bbcde75ab2cdb6a05: Initiating tablet copy from peer da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357)
I20240501 14:10:36.528240 23341 tablet_copy_client.cc:308] T bb291a0c743c4e638f79da36cc77faa3 P 778c78105092428bbcde75ab2cdb6a05: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.132:35357
I20240501 14:10:36.528388 21268 raft_consensus.cc:1261] T fe554c3145de46cf8070efcecffbaedb P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:36.528967 19789 tablet_copy_service.cc:140] P da3c4e948ff747678f18456c134d7ed4: Received BeginTabletCopySession request for tablet bb291a0c743c4e638f79da36cc77faa3 from peer 778c78105092428bbcde75ab2cdb6a05 ({username='slave'} at 127.7.39.136:40943)
I20240501 14:10:36.529050 19789 tablet_copy_service.cc:161] P da3c4e948ff747678f18456c134d7ed4: Beginning new tablet copy session on tablet bb291a0c743c4e638f79da36cc77faa3 from peer 778c78105092428bbcde75ab2cdb6a05 at {username='slave'} at 127.7.39.136:40943: session id = 778c78105092428bbcde75ab2cdb6a05-bb291a0c743c4e638f79da36cc77faa3
I20240501 14:10:36.529588 19227 catalog_manager.cc:5484] T dadab886b03540579d138cdb5b25e286 P 99a0be6ef2144b7fb247ba9d3f051d66 reported cstate change: config changed from index -1 to 4, VOTER 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136) added. New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:36.530184 19789 tablet_copy_source_session.cc:205] T bb291a0c743c4e638f79da36cc77faa3 P da3c4e948ff747678f18456c134d7ed4: Tablet Copy: opened 0 blocks and 1 log segments
W20240501 14:10:36.530447 19571 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136:34895): Couldn't send request to peer 778c78105092428bbcde75ab2cdb6a05. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fe554c3145de46cf8070efcecffbaedb. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:36.530675 19505 raft_consensus.cc:1261] T fe554c3145de46cf8070efcecffbaedb P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:36.531136 22458 consensus_queue.cc:1035] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:36.533116 22458 consensus_queue.cc:1035] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:36.533953 23341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb291a0c743c4e638f79da36cc77faa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:36.535286 22458 consensus_queue.cc:1035] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:36.536093 23341 tablet_copy_client.cc:783] T bb291a0c743c4e638f79da36cc77faa3 P 778c78105092428bbcde75ab2cdb6a05: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:36.536262 23341 tablet_copy_client.cc:647] T bb291a0c743c4e638f79da36cc77faa3 P 778c78105092428bbcde75ab2cdb6a05: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:36.537495 22086 raft_consensus.cc:2932] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:36.537873 19769 raft_consensus.cc:2932] T fe554c3145de46cf8070efcecffbaedb P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:36.540464 19234 catalog_manager.cc:5484] T fe554c3145de46cf8070efcecffbaedb P da3c4e948ff747678f18456c134d7ed4 reported cstate change: config changed from index 4 to 5, VOTER 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136) added. New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:36.543404 19505 raft_consensus.cc:2932] T fe554c3145de46cf8070efcecffbaedb P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:36.548192 23779 raft_consensus.cc:2932] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:36.550403 19505 raft_consensus.cc:2932] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:36.552381 23341 tablet_copy_client.cc:523] T bb291a0c743c4e638f79da36cc77faa3 P 778c78105092428bbcde75ab2cdb6a05: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:36.552752 19234 catalog_manager.cc:5484] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P fb1e6bc2d07b4ce3b622a66f14c93bdc reported cstate change: config changed from index -1 to 4, VOTER 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133) added. New cstate: current_term: 1 leader_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:36.553704 23341 tablet_bootstrap.cc:492] T bb291a0c743c4e638f79da36cc77faa3 P 778c78105092428bbcde75ab2cdb6a05: Bootstrap starting.
I20240501 14:10:36.554976 19769 raft_consensus.cc:2932] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:36.558476 21268 raft_consensus.cc:2932] T fe554c3145de46cf8070efcecffbaedb P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:36.585655 23341 tablet_bootstrap.cc:492] T bb291a0c743c4e638f79da36cc77faa3 P 778c78105092428bbcde75ab2cdb6a05: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:36.586038 23341 tablet_bootstrap.cc:492] T bb291a0c743c4e638f79da36cc77faa3 P 778c78105092428bbcde75ab2cdb6a05: Bootstrap complete.
I20240501 14:10:36.586220 23341 ts_tablet_manager.cc:1367] T bb291a0c743c4e638f79da36cc77faa3 P 778c78105092428bbcde75ab2cdb6a05: Time spent bootstrapping tablet: real 0.033s user 0.025s sys 0.000s
I20240501 14:10:36.586580 23341 raft_consensus.cc:348] T bb291a0c743c4e638f79da36cc77faa3 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:36.586679 23341 raft_consensus.cc:729] T bb291a0c743c4e638f79da36cc77faa3 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 778c78105092428bbcde75ab2cdb6a05, State: Initialized, Role: FOLLOWER
I20240501 14:10:36.586791 23341 consensus_queue.cc:260] T bb291a0c743c4e638f79da36cc77faa3 P 778c78105092428bbcde75ab2cdb6a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:36.587075 23341 ts_tablet_manager.cc:1398] T bb291a0c743c4e638f79da36cc77faa3 P 778c78105092428bbcde75ab2cdb6a05: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:36.587595 19789 tablet_copy_service.cc:342] P da3c4e948ff747678f18456c134d7ed4: Request end of tablet copy session 778c78105092428bbcde75ab2cdb6a05-bb291a0c743c4e638f79da36cc77faa3 received from {username='slave'} at 127.7.39.136:40943
I20240501 14:10:36.587688 19789 tablet_copy_service.cc:433] P da3c4e948ff747678f18456c134d7ed4: ending tablet copy session 778c78105092428bbcde75ab2cdb6a05-bb291a0c743c4e638f79da36cc77faa3 on tablet bb291a0c743c4e638f79da36cc77faa3 with peer 778c78105092428bbcde75ab2cdb6a05
I20240501 14:10:36.619109 21136 raft_consensus.cc:1203] T 1d408b075d784953a57554bcdf19158a P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:10:36.647796 21398 raft_consensus.cc:1203] T cc8e888d46d34a22b6d4409ec2ab53a9 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:10:36.653086 21268 raft_consensus.cc:1203] T 2dcca6e0e6374520bd4522c80c110e97 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:10:36.711364 21268 raft_consensus.cc:1203] T d63afd081986473aa2715d074b123a93 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:10:36.717779 23632 ts_tablet_manager.cc:902] T 661aaf83e3474966a257fcc039ef4784 P e6246b183293412494919f1e7efc9448: Initiating tablet copy from peer ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201)
I20240501 14:10:36.718331 23632 tablet_copy_client.cc:308] T 661aaf83e3474966a257fcc039ef4784 P e6246b183293412494919f1e7efc9448: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.131:45201
I20240501 14:10:36.718757 19657 tablet_copy_service.cc:140] P ec8cde17a00645e7b38d5d29f7425367: Received BeginTabletCopySession request for tablet 661aaf83e3474966a257fcc039ef4784 from peer e6246b183293412494919f1e7efc9448 ({username='slave'} at 127.7.39.134:41913)
I20240501 14:10:36.718861 19657 tablet_copy_service.cc:161] P ec8cde17a00645e7b38d5d29f7425367: Beginning new tablet copy session on tablet 661aaf83e3474966a257fcc039ef4784 from peer e6246b183293412494919f1e7efc9448 at {username='slave'} at 127.7.39.134:41913: session id = e6246b183293412494919f1e7efc9448-661aaf83e3474966a257fcc039ef4784
I20240501 14:10:36.719092 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:36.719882 19657 tablet_copy_source_session.cc:205] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:36.719913 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:36.720642 23632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 661aaf83e3474966a257fcc039ef4784. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:36.722761 23632 tablet_copy_client.cc:783] T 661aaf83e3474966a257fcc039ef4784 P e6246b183293412494919f1e7efc9448: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:36.722934 23632 tablet_copy_client.cc:647] T 661aaf83e3474966a257fcc039ef4784 P e6246b183293412494919f1e7efc9448: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:36.723783 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:36.725989 23632 tablet_copy_client.cc:523] T 661aaf83e3474966a257fcc039ef4784 P e6246b183293412494919f1e7efc9448: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:36.727447 23632 tablet_bootstrap.cc:492] T 661aaf83e3474966a257fcc039ef4784 P e6246b183293412494919f1e7efc9448: Bootstrap starting.
I20240501 14:10:36.735935 21247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:36.742545 21530 raft_consensus.cc:1203] T ec2a46f8cfb3406082970e5fdb59b101 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:10:36.749902 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:36.770726 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:36.772858 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:36.784435 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:36.821555 21136 raft_consensus.cc:1203] T bb291a0c743c4e638f79da36cc77faa3 P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:10:36.834522 21398 raft_consensus.cc:1203] T f6ca088ce3aa4b97aa54c8ce6d700921 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:10:36.848279 21268 raft_consensus.cc:1203] T cc8e888d46d34a22b6d4409ec2ab53a9 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:10:36.851153 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:36.852521 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:36.853154 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:36.864815 23632 tablet_bootstrap.cc:492] T 661aaf83e3474966a257fcc039ef4784 P e6246b183293412494919f1e7efc9448: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:36.866897 23558 ts_tablet_manager.cc:902] T 12ffa74f7a5b472aac57253693273cad P e6246b183293412494919f1e7efc9448: Initiating tablet copy from peer ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201)
I20240501 14:10:36.867511 23558 tablet_copy_client.cc:308] T 12ffa74f7a5b472aac57253693273cad P e6246b183293412494919f1e7efc9448: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.131:45201
I20240501 14:10:36.867997 19657 tablet_copy_service.cc:140] P ec8cde17a00645e7b38d5d29f7425367: Received BeginTabletCopySession request for tablet 12ffa74f7a5b472aac57253693273cad from peer e6246b183293412494919f1e7efc9448 ({username='slave'} at 127.7.39.134:41913)
I20240501 14:10:36.868114 19657 tablet_copy_service.cc:161] P ec8cde17a00645e7b38d5d29f7425367: Beginning new tablet copy session on tablet 12ffa74f7a5b472aac57253693273cad from peer e6246b183293412494919f1e7efc9448 at {username='slave'} at 127.7.39.134:41913: session id = e6246b183293412494919f1e7efc9448-12ffa74f7a5b472aac57253693273cad
I20240501 14:10:36.869135 19657 tablet_copy_source_session.cc:205] T 12ffa74f7a5b472aac57253693273cad P ec8cde17a00645e7b38d5d29f7425367: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:36.870172 23558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12ffa74f7a5b472aac57253693273cad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:36.879336 21247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:36.881953 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:36.883313 23632 tablet_bootstrap.cc:492] T 661aaf83e3474966a257fcc039ef4784 P e6246b183293412494919f1e7efc9448: Bootstrap complete.
I20240501 14:10:36.883531 23632 ts_tablet_manager.cc:1367] T 661aaf83e3474966a257fcc039ef4784 P e6246b183293412494919f1e7efc9448: Time spent bootstrapping tablet: real 0.156s user 0.021s sys 0.007s
I20240501 14:10:36.884014 23632 raft_consensus.cc:348] T 661aaf83e3474966a257fcc039ef4784 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:36.884135 23632 raft_consensus.cc:729] T 661aaf83e3474966a257fcc039ef4784 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6246b183293412494919f1e7efc9448, State: Initialized, Role: FOLLOWER
I20240501 14:10:36.884255 23632 consensus_queue.cc:260] T 661aaf83e3474966a257fcc039ef4784 P e6246b183293412494919f1e7efc9448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:36.884508 23632 ts_tablet_manager.cc:1398] T 661aaf83e3474966a257fcc039ef4784 P e6246b183293412494919f1e7efc9448: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:36.886677 23558 tablet_copy_client.cc:783] T 12ffa74f7a5b472aac57253693273cad P e6246b183293412494919f1e7efc9448: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:36.886837 23558 tablet_copy_client.cc:647] T 12ffa74f7a5b472aac57253693273cad P e6246b183293412494919f1e7efc9448: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:36.887596 19657 tablet_copy_service.cc:342] P ec8cde17a00645e7b38d5d29f7425367: Request end of tablet copy session e6246b183293412494919f1e7efc9448-661aaf83e3474966a257fcc039ef4784 received from {username='slave'} at 127.7.39.134:41913
I20240501 14:10:36.887679 19657 tablet_copy_service.cc:433] P ec8cde17a00645e7b38d5d29f7425367: ending tablet copy session e6246b183293412494919f1e7efc9448-661aaf83e3474966a257fcc039ef4784 on tablet 661aaf83e3474966a257fcc039ef4784 with peer e6246b183293412494919f1e7efc9448
I20240501 14:10:36.889365 23558 tablet_copy_client.cc:523] T 12ffa74f7a5b472aac57253693273cad P e6246b183293412494919f1e7efc9448: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:36.890719 23558 tablet_bootstrap.cc:492] T 12ffa74f7a5b472aac57253693273cad P e6246b183293412494919f1e7efc9448: Bootstrap starting.
I20240501 14:10:36.906352 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:36.910224 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:36.912859 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:36.919524 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:36.919850 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:36.935609 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:36.935992 21247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:36.940191 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:36.940832 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:36.941251 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:36.952239 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:36.954070 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:36.966615 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:36.974161 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:36.978838 21530 raft_consensus.cc:1203] T bb291a0c743c4e638f79da36cc77faa3 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:10:36.986119 21247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:37.002791 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:37.010092 23341 ts_tablet_manager.cc:902] T fe554c3145de46cf8070efcecffbaedb P 778c78105092428bbcde75ab2cdb6a05: Initiating tablet copy from peer ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201)
I20240501 14:10:37.010557 23341 tablet_copy_client.cc:308] T fe554c3145de46cf8070efcecffbaedb P 778c78105092428bbcde75ab2cdb6a05: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.131:45201
I20240501 14:10:37.011366 19657 tablet_copy_service.cc:140] P ec8cde17a00645e7b38d5d29f7425367: Received BeginTabletCopySession request for tablet fe554c3145de46cf8070efcecffbaedb from peer 778c78105092428bbcde75ab2cdb6a05 ({username='slave'} at 127.7.39.136:34889)
I20240501 14:10:37.011480 19657 tablet_copy_service.cc:161] P ec8cde17a00645e7b38d5d29f7425367: Beginning new tablet copy session on tablet fe554c3145de46cf8070efcecffbaedb from peer 778c78105092428bbcde75ab2cdb6a05 at {username='slave'} at 127.7.39.136:34889: session id = 778c78105092428bbcde75ab2cdb6a05-fe554c3145de46cf8070efcecffbaedb
I20240501 14:10:37.012480 19657 tablet_copy_source_session.cc:205] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:37.014426 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:37.022697 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:37.028252 23341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe554c3145de46cf8070efcecffbaedb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:37.031024 23341 tablet_copy_client.cc:783] T fe554c3145de46cf8070efcecffbaedb P 778c78105092428bbcde75ab2cdb6a05: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:37.031315 23341 tablet_copy_client.cc:647] T fe554c3145de46cf8070efcecffbaedb P 778c78105092428bbcde75ab2cdb6a05: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:37.046397 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:37.047057 24041 ts_tablet_manager.cc:902] T dab5e5722e2e4b0eaea73288d8dce265 P 5bf81f3d986e428190d3f7727f36d96c: Initiating tablet copy from peer da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357)
I20240501 14:10:37.047585 24041 tablet_copy_client.cc:308] T dab5e5722e2e4b0eaea73288d8dce265 P 5bf81f3d986e428190d3f7727f36d96c: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.132:35357
I20240501 14:10:37.048040 19789 tablet_copy_service.cc:140] P da3c4e948ff747678f18456c134d7ed4: Received BeginTabletCopySession request for tablet dab5e5722e2e4b0eaea73288d8dce265 from peer 5bf81f3d986e428190d3f7727f36d96c ({username='slave'} at 127.7.39.133:51567)
I20240501 14:10:37.048163 19789 tablet_copy_service.cc:161] P da3c4e948ff747678f18456c134d7ed4: Beginning new tablet copy session on tablet dab5e5722e2e4b0eaea73288d8dce265 from peer 5bf81f3d986e428190d3f7727f36d96c at {username='slave'} at 127.7.39.133:51567: session id = 5bf81f3d986e428190d3f7727f36d96c-dab5e5722e2e4b0eaea73288d8dce265
I20240501 14:10:37.049321 19789 tablet_copy_source_session.cc:205] T dab5e5722e2e4b0eaea73288d8dce265 P da3c4e948ff747678f18456c134d7ed4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:37.051368 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:37.054917 21247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:37.057866 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:37.064538 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:37.074771 24041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dab5e5722e2e4b0eaea73288d8dce265. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:37.077406 24041 tablet_copy_client.cc:783] T dab5e5722e2e4b0eaea73288d8dce265 P 5bf81f3d986e428190d3f7727f36d96c: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:37.077621 24041 tablet_copy_client.cc:647] T dab5e5722e2e4b0eaea73288d8dce265 P 5bf81f3d986e428190d3f7727f36d96c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:37.082861 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:37.084412 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:37.087980 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:37.096490 24041 tablet_copy_client.cc:523] T dab5e5722e2e4b0eaea73288d8dce265 P 5bf81f3d986e428190d3f7727f36d96c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:37.098011 24041 tablet_bootstrap.cc:492] T dab5e5722e2e4b0eaea73288d8dce265 P 5bf81f3d986e428190d3f7727f36d96c: Bootstrap starting.
I20240501 14:10:37.106664 24050 ts_tablet_manager.cc:902] T dadab886b03540579d138cdb5b25e286 P 778c78105092428bbcde75ab2cdb6a05: Initiating tablet copy from peer ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201)
I20240501 14:10:37.107300 24050 tablet_copy_client.cc:308] T dadab886b03540579d138cdb5b25e286 P 778c78105092428bbcde75ab2cdb6a05: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.131:45201
I20240501 14:10:37.107831 19657 tablet_copy_service.cc:140] P ec8cde17a00645e7b38d5d29f7425367: Received BeginTabletCopySession request for tablet dadab886b03540579d138cdb5b25e286 from peer 778c78105092428bbcde75ab2cdb6a05 ({username='slave'} at 127.7.39.136:34889)
I20240501 14:10:37.107934 19657 tablet_copy_service.cc:161] P ec8cde17a00645e7b38d5d29f7425367: Beginning new tablet copy session on tablet dadab886b03540579d138cdb5b25e286 from peer 778c78105092428bbcde75ab2cdb6a05 at {username='slave'} at 127.7.39.136:34889: session id = 778c78105092428bbcde75ab2cdb6a05-dadab886b03540579d138cdb5b25e286
I20240501 14:10:37.107980 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:37.108891 19657 tablet_copy_source_session.cc:205] T dadab886b03540579d138cdb5b25e286 P ec8cde17a00645e7b38d5d29f7425367: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:37.109894 24050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dadab886b03540579d138cdb5b25e286. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:37.112766 24050 tablet_copy_client.cc:783] T dadab886b03540579d138cdb5b25e286 P 778c78105092428bbcde75ab2cdb6a05: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:37.112967 24050 tablet_copy_client.cc:647] T dadab886b03540579d138cdb5b25e286 P 778c78105092428bbcde75ab2cdb6a05: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:37.114012 24050 tablet_copy_client.cc:523] T dadab886b03540579d138cdb5b25e286 P 778c78105092428bbcde75ab2cdb6a05: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:37.115299 24050 tablet_bootstrap.cc:492] T dadab886b03540579d138cdb5b25e286 P 778c78105092428bbcde75ab2cdb6a05: Bootstrap starting.
I20240501 14:10:37.128005 23341 tablet_copy_client.cc:523] T fe554c3145de46cf8070efcecffbaedb P 778c78105092428bbcde75ab2cdb6a05: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:37.129477 23341 tablet_bootstrap.cc:492] T fe554c3145de46cf8070efcecffbaedb P 778c78105092428bbcde75ab2cdb6a05: Bootstrap starting.
I20240501 14:10:37.144511 24058 ts_tablet_manager.cc:902] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P 5bf81f3d986e428190d3f7727f36d96c: Initiating tablet copy from peer fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847)
I20240501 14:10:37.145068 24058 tablet_copy_client.cc:308] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P 5bf81f3d986e428190d3f7727f36d96c: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.129:35847
I20240501 14:10:37.145504 19396 tablet_copy_service.cc:140] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Received BeginTabletCopySession request for tablet 78e14dea8b6e4c0abd1c88dcd5b4e608 from peer 5bf81f3d986e428190d3f7727f36d96c ({username='slave'} at 127.7.39.133:53931)
I20240501 14:10:37.145625 19396 tablet_copy_service.cc:161] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Beginning new tablet copy session on tablet 78e14dea8b6e4c0abd1c88dcd5b4e608 from peer 5bf81f3d986e428190d3f7727f36d96c at {username='slave'} at 127.7.39.133:53931: session id = 5bf81f3d986e428190d3f7727f36d96c-78e14dea8b6e4c0abd1c88dcd5b4e608
I20240501 14:10:37.146529 19396 tablet_copy_source_session.cc:205] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:37.147513 24058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78e14dea8b6e4c0abd1c88dcd5b4e608. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:37.150437 24058 tablet_copy_client.cc:783] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P 5bf81f3d986e428190d3f7727f36d96c: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:37.150787 24058 tablet_copy_client.cc:647] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P 5bf81f3d986e428190d3f7727f36d96c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:37.151847 24058 tablet_copy_client.cc:523] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P 5bf81f3d986e428190d3f7727f36d96c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:37.152809 24050 tablet_bootstrap.cc:492] T dadab886b03540579d138cdb5b25e286 P 778c78105092428bbcde75ab2cdb6a05: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:37.153015 24058 tablet_bootstrap.cc:492] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P 5bf81f3d986e428190d3f7727f36d96c: Bootstrap starting.
I20240501 14:10:37.153193 24050 tablet_bootstrap.cc:492] T dadab886b03540579d138cdb5b25e286 P 778c78105092428bbcde75ab2cdb6a05: Bootstrap complete.
I20240501 14:10:37.153395 24050 ts_tablet_manager.cc:1367] T dadab886b03540579d138cdb5b25e286 P 778c78105092428bbcde75ab2cdb6a05: Time spent bootstrapping tablet: real 0.038s user 0.023s sys 0.003s
I20240501 14:10:37.153892 24050 raft_consensus.cc:348] T dadab886b03540579d138cdb5b25e286 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:10:37.154083 24050 raft_consensus.cc:729] T dadab886b03540579d138cdb5b25e286 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 778c78105092428bbcde75ab2cdb6a05, State: Initialized, Role: FOLLOWER
I20240501 14:10:37.154332 24050 consensus_queue.cc:260] T dadab886b03540579d138cdb5b25e286 P 778c78105092428bbcde75ab2cdb6a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:10:37.155308 24050 ts_tablet_manager.cc:1398] T dadab886b03540579d138cdb5b25e286 P 778c78105092428bbcde75ab2cdb6a05: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:10:37.155841 19657 tablet_copy_service.cc:342] P ec8cde17a00645e7b38d5d29f7425367: Request end of tablet copy session 778c78105092428bbcde75ab2cdb6a05-dadab886b03540579d138cdb5b25e286 received from {username='slave'} at 127.7.39.136:34889
I20240501 14:10:37.155927 19657 tablet_copy_service.cc:433] P ec8cde17a00645e7b38d5d29f7425367: ending tablet copy session 778c78105092428bbcde75ab2cdb6a05-dadab886b03540579d138cdb5b25e286 on tablet dadab886b03540579d138cdb5b25e286 with peer 778c78105092428bbcde75ab2cdb6a05
I20240501 14:10:37.163012 23558 tablet_bootstrap.cc:492] T 12ffa74f7a5b472aac57253693273cad P e6246b183293412494919f1e7efc9448: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:37.163887 23558 tablet_bootstrap.cc:492] T 12ffa74f7a5b472aac57253693273cad P e6246b183293412494919f1e7efc9448: Bootstrap complete.
I20240501 14:10:37.164075 23558 ts_tablet_manager.cc:1367] T 12ffa74f7a5b472aac57253693273cad P e6246b183293412494919f1e7efc9448: Time spent bootstrapping tablet: real 0.273s user 0.022s sys 0.004s
I20240501 14:10:37.164512 23558 raft_consensus.cc:348] T 12ffa74f7a5b472aac57253693273cad P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:37.164642 23558 raft_consensus.cc:729] T 12ffa74f7a5b472aac57253693273cad P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6246b183293412494919f1e7efc9448, State: Initialized, Role: FOLLOWER
I20240501 14:10:37.164808 23558 consensus_queue.cc:260] T 12ffa74f7a5b472aac57253693273cad P e6246b183293412494919f1e7efc9448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:37.166966 23558 ts_tablet_manager.cc:1398] T 12ffa74f7a5b472aac57253693273cad P e6246b183293412494919f1e7efc9448: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:10:37.168808 19657 tablet_copy_service.cc:342] P ec8cde17a00645e7b38d5d29f7425367: Request end of tablet copy session e6246b183293412494919f1e7efc9448-12ffa74f7a5b472aac57253693273cad received from {username='slave'} at 127.7.39.134:41913
I20240501 14:10:37.168892 19657 tablet_copy_service.cc:433] P ec8cde17a00645e7b38d5d29f7425367: ending tablet copy session e6246b183293412494919f1e7efc9448-12ffa74f7a5b472aac57253693273cad on tablet 12ffa74f7a5b472aac57253693273cad with peer e6246b183293412494919f1e7efc9448
I20240501 14:10:37.191658 24041 tablet_bootstrap.cc:492] T dab5e5722e2e4b0eaea73288d8dce265 P 5bf81f3d986e428190d3f7727f36d96c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:37.192080 24041 tablet_bootstrap.cc:492] T dab5e5722e2e4b0eaea73288d8dce265 P 5bf81f3d986e428190d3f7727f36d96c: Bootstrap complete.
I20240501 14:10:37.192344 24041 ts_tablet_manager.cc:1367] T dab5e5722e2e4b0eaea73288d8dce265 P 5bf81f3d986e428190d3f7727f36d96c: Time spent bootstrapping tablet: real 0.094s user 0.021s sys 0.006s
I20240501 14:10:37.192867 24058 tablet_bootstrap.cc:492] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P 5bf81f3d986e428190d3f7727f36d96c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:37.192859 24041 raft_consensus.cc:348] T dab5e5722e2e4b0eaea73288d8dce265 P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:37.193125 24041 raft_consensus.cc:729] T dab5e5722e2e4b0eaea73288d8dce265 P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf81f3d986e428190d3f7727f36d96c, State: Initialized, Role: FOLLOWER
I20240501 14:10:37.193189 24058 tablet_bootstrap.cc:492] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P 5bf81f3d986e428190d3f7727f36d96c: Bootstrap complete.
I20240501 14:10:37.193354 24041 consensus_queue.cc:260] T dab5e5722e2e4b0eaea73288d8dce265 P 5bf81f3d986e428190d3f7727f36d96c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:37.193424 24058 ts_tablet_manager.cc:1367] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P 5bf81f3d986e428190d3f7727f36d96c: Time spent bootstrapping tablet: real 0.040s user 0.023s sys 0.004s
I20240501 14:10:37.193804 24041 ts_tablet_manager.cc:1398] T dab5e5722e2e4b0eaea73288d8dce265 P 5bf81f3d986e428190d3f7727f36d96c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:37.193895 24058 raft_consensus.cc:348] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:37.194013 24058 raft_consensus.cc:729] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf81f3d986e428190d3f7727f36d96c, State: Initialized, Role: FOLLOWER
I20240501 14:10:37.194128 24058 consensus_queue.cc:260] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P 5bf81f3d986e428190d3f7727f36d96c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:37.194308 19789 tablet_copy_service.cc:342] P da3c4e948ff747678f18456c134d7ed4: Request end of tablet copy session 5bf81f3d986e428190d3f7727f36d96c-dab5e5722e2e4b0eaea73288d8dce265 received from {username='slave'} at 127.7.39.133:51567
I20240501 14:10:37.194392 19789 tablet_copy_service.cc:433] P da3c4e948ff747678f18456c134d7ed4: ending tablet copy session 5bf81f3d986e428190d3f7727f36d96c-dab5e5722e2e4b0eaea73288d8dce265 on tablet dab5e5722e2e4b0eaea73288d8dce265 with peer 5bf81f3d986e428190d3f7727f36d96c
I20240501 14:10:37.194432 24058 ts_tablet_manager.cc:1398] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P 5bf81f3d986e428190d3f7727f36d96c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:37.195209 19396 tablet_copy_service.cc:342] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Request end of tablet copy session 5bf81f3d986e428190d3f7727f36d96c-78e14dea8b6e4c0abd1c88dcd5b4e608 received from {username='slave'} at 127.7.39.133:53931
I20240501 14:10:37.195278 19396 tablet_copy_service.cc:433] P fb1e6bc2d07b4ce3b622a66f14c93bdc: ending tablet copy session 5bf81f3d986e428190d3f7727f36d96c-78e14dea8b6e4c0abd1c88dcd5b4e608 on tablet 78e14dea8b6e4c0abd1c88dcd5b4e608 with peer 5bf81f3d986e428190d3f7727f36d96c
I20240501 14:10:37.207201 23341 tablet_bootstrap.cc:492] T fe554c3145de46cf8070efcecffbaedb P 778c78105092428bbcde75ab2cdb6a05: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:37.207543 23341 tablet_bootstrap.cc:492] T fe554c3145de46cf8070efcecffbaedb P 778c78105092428bbcde75ab2cdb6a05: Bootstrap complete.
I20240501 14:10:37.207731 23341 ts_tablet_manager.cc:1367] T fe554c3145de46cf8070efcecffbaedb P 778c78105092428bbcde75ab2cdb6a05: Time spent bootstrapping tablet: real 0.078s user 0.024s sys 0.001s
I20240501 14:10:37.208181 23341 raft_consensus.cc:348] T fe554c3145de46cf8070efcecffbaedb P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:10:37.208317 23341 raft_consensus.cc:729] T fe554c3145de46cf8070efcecffbaedb P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 778c78105092428bbcde75ab2cdb6a05, State: Initialized, Role: FOLLOWER
I20240501 14:10:37.208467 23341 consensus_queue.cc:260] T fe554c3145de46cf8070efcecffbaedb P 778c78105092428bbcde75ab2cdb6a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:10:37.208787 23341 ts_tablet_manager.cc:1398] T fe554c3145de46cf8070efcecffbaedb P 778c78105092428bbcde75ab2cdb6a05: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:37.209357 19657 tablet_copy_service.cc:342] P ec8cde17a00645e7b38d5d29f7425367: Request end of tablet copy session 778c78105092428bbcde75ab2cdb6a05-fe554c3145de46cf8070efcecffbaedb received from {username='slave'} at 127.7.39.136:34889
I20240501 14:10:37.209470 19657 tablet_copy_service.cc:433] P ec8cde17a00645e7b38d5d29f7425367: ending tablet copy session 778c78105092428bbcde75ab2cdb6a05-fe554c3145de46cf8070efcecffbaedb on tablet fe554c3145de46cf8070efcecffbaedb with peer 778c78105092428bbcde75ab2cdb6a05
I20240501 14:10:37.264518 21268 raft_consensus.cc:1203] T 661aaf83e3474966a257fcc039ef4784 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:10:37.276895 19636 consensus_queue.cc:237] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:37.278510 19506 raft_consensus.cc:1261] T ffb4b6a2c19b497f8117fbbc9ea50082 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:10:37.278764 19375 raft_consensus.cc:1261] T ffb4b6a2c19b497f8117fbbc9ea50082 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:10:37.278993 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ffb4b6a2c19b497f8117fbbc9ea50082. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:37.279330 22458 consensus_queue.cc:1035] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:37.279673 24190 consensus_queue.cc:1035] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:37.284955 22580 raft_consensus.cc:2932] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:37.286790 19506 raft_consensus.cc:2932] T ffb4b6a2c19b497f8117fbbc9ea50082 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:37.290721 19225 catalog_manager.cc:5484] T ffb4b6a2c19b497f8117fbbc9ea50082 P 99a0be6ef2144b7fb247ba9d3f051d66 reported cstate change: config changed from index -1 to 4, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:37.292749 19375 raft_consensus.cc:2932] T ffb4b6a2c19b497f8117fbbc9ea50082 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:37.429019 21530 raft_consensus.cc:1203] T fe554c3145de46cf8070efcecffbaedb P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:10:37.439288 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:37.451867 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:37.452701 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:37.455861 21247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:37.467248 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:37.473662 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:37.474023 21268 raft_consensus.cc:1203] T 12ffa74f7a5b472aac57253693273cad P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:10:37.476187 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:37.488710 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:37.490226 21136 raft_consensus.cc:1203] T dab5e5722e2e4b0eaea73288d8dce265 P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:10:37.493522 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:37.508082 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:37.508730 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:37.518036 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:37.521203 21530 raft_consensus.cc:1203] T dadab886b03540579d138cdb5b25e286 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:10:37.523221 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:37.537983 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:37.550202 21247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:37.585322 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:37.589016 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:37.589084 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:37.590047 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:37.597465 21136 raft_consensus.cc:1203] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:10:37.603801 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:37.617821 21247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:37.625208 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:37.636690 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:37.645913 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:37.649551 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:37.654232 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:37.681517 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:37.688177 21247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:37.694032 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:37.707881 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:37.717792 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:37.724292 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:37.732350 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:37.738934 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:37.750900 21247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:37.764914 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:37.771261 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:37.780983 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:37.803356 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:37.805553 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:37.874248 24434 ts_tablet_manager.cc:902] T ffb4b6a2c19b497f8117fbbc9ea50082 P e6246b183293412494919f1e7efc9448: Initiating tablet copy from peer ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201)
I20240501 14:10:37.875360 24434 tablet_copy_client.cc:308] T ffb4b6a2c19b497f8117fbbc9ea50082 P e6246b183293412494919f1e7efc9448: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.131:45201
I20240501 14:10:37.875898 19657 tablet_copy_service.cc:140] P ec8cde17a00645e7b38d5d29f7425367: Received BeginTabletCopySession request for tablet ffb4b6a2c19b497f8117fbbc9ea50082 from peer e6246b183293412494919f1e7efc9448 ({username='slave'} at 127.7.39.134:41913)
I20240501 14:10:37.876039 19657 tablet_copy_service.cc:161] P ec8cde17a00645e7b38d5d29f7425367: Beginning new tablet copy session on tablet ffb4b6a2c19b497f8117fbbc9ea50082 from peer e6246b183293412494919f1e7efc9448 at {username='slave'} at 127.7.39.134:41913: session id = e6246b183293412494919f1e7efc9448-ffb4b6a2c19b497f8117fbbc9ea50082
I20240501 14:10:37.877166 19657 tablet_copy_source_session.cc:205] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:37.881069 24434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffb4b6a2c19b497f8117fbbc9ea50082. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:37.884070 24434 tablet_copy_client.cc:783] T ffb4b6a2c19b497f8117fbbc9ea50082 P e6246b183293412494919f1e7efc9448: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:37.884244 24434 tablet_copy_client.cc:647] T ffb4b6a2c19b497f8117fbbc9ea50082 P e6246b183293412494919f1e7efc9448: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:37.885267 24434 tablet_copy_client.cc:523] T ffb4b6a2c19b497f8117fbbc9ea50082 P e6246b183293412494919f1e7efc9448: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:37.886505 24434 tablet_bootstrap.cc:492] T ffb4b6a2c19b497f8117fbbc9ea50082 P e6246b183293412494919f1e7efc9448: Bootstrap starting.
I20240501 14:10:37.928231 24434 tablet_bootstrap.cc:492] T ffb4b6a2c19b497f8117fbbc9ea50082 P e6246b183293412494919f1e7efc9448: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:37.928623 24434 tablet_bootstrap.cc:492] T ffb4b6a2c19b497f8117fbbc9ea50082 P e6246b183293412494919f1e7efc9448: Bootstrap complete.
I20240501 14:10:37.928704 19376 consensus_queue.cc:237] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:37.928803 24434 ts_tablet_manager.cc:1367] T ffb4b6a2c19b497f8117fbbc9ea50082 P e6246b183293412494919f1e7efc9448: Time spent bootstrapping tablet: real 0.042s user 0.028s sys 0.000s
I20240501 14:10:37.929217 24434 raft_consensus.cc:348] T ffb4b6a2c19b497f8117fbbc9ea50082 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:37.929317 24434 raft_consensus.cc:729] T ffb4b6a2c19b497f8117fbbc9ea50082 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6246b183293412494919f1e7efc9448, State: Initialized, Role: FOLLOWER
I20240501 14:10:37.929461 24434 consensus_queue.cc:260] T ffb4b6a2c19b497f8117fbbc9ea50082 P e6246b183293412494919f1e7efc9448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:37.930426 24434 ts_tablet_manager.cc:1398] T ffb4b6a2c19b497f8117fbbc9ea50082 P e6246b183293412494919f1e7efc9448: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:10:37.930555 19506 raft_consensus.cc:1261] T f69020bb6c244848a6b0a50fd2b4f3eb P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:37.930953 19657 tablet_copy_service.cc:342] P ec8cde17a00645e7b38d5d29f7425367: Request end of tablet copy session e6246b183293412494919f1e7efc9448-ffb4b6a2c19b497f8117fbbc9ea50082 received from {username='slave'} at 127.7.39.134:41913
I20240501 14:10:37.931089 19657 tablet_copy_service.cc:433] P ec8cde17a00645e7b38d5d29f7425367: ending tablet copy session e6246b183293412494919f1e7efc9448-ffb4b6a2c19b497f8117fbbc9ea50082 on tablet ffb4b6a2c19b497f8117fbbc9ea50082 with peer e6246b183293412494919f1e7efc9448
I20240501 14:10:37.931617 21136 raft_consensus.cc:1261] T f69020bb6c244848a6b0a50fd2b4f3eb P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:37.932294 23779 consensus_queue.cc:1035] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:37.933936 23779 consensus_queue.cc:1035] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:37.936434 23779 raft_consensus.cc:2932] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:37.938028 19769 consensus_queue.cc:237] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:37.939635 21268 raft_consensus.cc:1261] T 11f64b59f83e458f8cf51fd5bcd36fee P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:37.939997 19636 raft_consensus.cc:1261] T 11f64b59f83e458f8cf51fd5bcd36fee P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:37.940627 24084 consensus_queue.cc:1035] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:37.940985 24084 consensus_queue.cc:1035] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:37.941277 19506 raft_consensus.cc:2932] T f69020bb6c244848a6b0a50fd2b4f3eb P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:37.941855 21136 raft_consensus.cc:2932] T f69020bb6c244848a6b0a50fd2b4f3eb P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:37.944670 19226 catalog_manager.cc:5484] T f69020bb6c244848a6b0a50fd2b4f3eb P 99a0be6ef2144b7fb247ba9d3f051d66 reported cstate change: config changed from index 4 to 5, VOTER ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131) evicted. New cstate: current_term: 1 leader_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:37.950547 24191 raft_consensus.cc:2932] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:37.953853 21268 raft_consensus.cc:2932] T 11f64b59f83e458f8cf51fd5bcd36fee P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:37.956924 19228 catalog_manager.cc:5484] T 11f64b59f83e458f8cf51fd5bcd36fee P e6246b183293412494919f1e7efc9448 reported cstate change: config changed from index 4 to 5, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) evicted. New cstate: current_term: 1 leader_uuid: "da3c4e948ff747678f18456c134d7ed4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:37.964260 19636 raft_consensus.cc:2932] T 11f64b59f83e458f8cf51fd5bcd36fee P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:37.987474 19617 tablet_service.cc:1514] Processing DeleteTablet for tablet f69020bb6c244848a6b0a50fd2b4f3eb with delete_type TABLET_DATA_TOMBSTONED (TS ec8cde17a00645e7b38d5d29f7425367 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:47238
I20240501 14:10:37.987915 19356 tablet_service.cc:1514] Processing DeleteTablet for tablet 11f64b59f83e458f8cf51fd5bcd36fee with delete_type TABLET_DATA_TOMBSTONED (TS fb1e6bc2d07b4ce3b622a66f14c93bdc not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:46838
I20240501 14:10:37.990537 24576 tablet_replica.cc:330] T 11f64b59f83e458f8cf51fd5bcd36fee P fb1e6bc2d07b4ce3b622a66f14c93bdc: stopping tablet replica
I20240501 14:10:37.994539 24576 raft_consensus.cc:2229] T 11f64b59f83e458f8cf51fd5bcd36fee P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:37.994974 24576 raft_consensus.cc:2258] T 11f64b59f83e458f8cf51fd5bcd36fee P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:37.996202 24576 ts_tablet_manager.cc:1875] T 11f64b59f83e458f8cf51fd5bcd36fee P fb1e6bc2d07b4ce3b622a66f14c93bdc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:37.998903 24576 ts_tablet_manager.cc:1888] T 11f64b59f83e458f8cf51fd5bcd36fee P fb1e6bc2d07b4ce3b622a66f14c93bdc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:10:37.999078 24576 log.cc:1196] T 11f64b59f83e458f8cf51fd5bcd36fee P fb1e6bc2d07b4ce3b622a66f14c93bdc: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal/wals/11f64b59f83e458f8cf51fd5bcd36fee
I20240501 14:10:37.999816 19218 catalog_manager.cc:4830] TS fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847): tablet 11f64b59f83e458f8cf51fd5bcd36fee (table rebalance_test_table_8 [id=3866c35dd0e64d87b6049e5e14cb0d31]) successfully deleted
I20240501 14:10:38.001379 24575 tablet_replica.cc:330] T f69020bb6c244848a6b0a50fd2b4f3eb P ec8cde17a00645e7b38d5d29f7425367: stopping tablet replica
I20240501 14:10:38.001701 24575 raft_consensus.cc:2229] T f69020bb6c244848a6b0a50fd2b4f3eb P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:38.001940 24575 raft_consensus.cc:2258] T f69020bb6c244848a6b0a50fd2b4f3eb P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:38.003479 24575 ts_tablet_manager.cc:1875] T f69020bb6c244848a6b0a50fd2b4f3eb P ec8cde17a00645e7b38d5d29f7425367: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:38.033159 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:38.036545 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:38.041273 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:38.052774 21247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:38.055480 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:38.057940 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:38.059650 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:38.071520 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:38.074143 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:38.080166 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:38.085942 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:38.125533 24575 ts_tablet_manager.cc:1888] T f69020bb6c244848a6b0a50fd2b4f3eb P ec8cde17a00645e7b38d5d29f7425367: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:10:38.125748 24575 log.cc:1196] T f69020bb6c244848a6b0a50fd2b4f3eb P ec8cde17a00645e7b38d5d29f7425367: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal/wals/f69020bb6c244848a6b0a50fd2b4f3eb
I20240501 14:10:38.127458 19219 catalog_manager.cc:4830] TS ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201): tablet f69020bb6c244848a6b0a50fd2b4f3eb (table rebalance_test_table_4 [id=151dfad0ef40470ab9c5203c42400b60]) successfully deleted
I20240501 14:10:38.127574 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:38.141932 21247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:38.159865 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:38.169337 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:38.176955 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:38.179198 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:38.189050 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:38.202030 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:38.211748 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:38.225701 21247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:38.235597 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:38.249120 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:38.249521 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:38.278410 21268 raft_consensus.cc:1203] T ffb4b6a2c19b497f8117fbbc9ea50082 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:10:38.306553 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:38.347734 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:38.357223 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:38.358944 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:38.362619 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:38.363809 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:38.371208 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:38.376369 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:38.384750 21247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:38.390403 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:38.393916 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:38.404135 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:38.445134 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:38.459722 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:38.465684 21247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:38.466778 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:38.497341 19636 consensus_queue.cc:237] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:38.499019 19376 raft_consensus.cc:1261] T 85593473500b4d5eaf9f8b0bf2296777 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:38.499481 22458 consensus_queue.cc:1035] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:38.500794 21268 raft_consensus.cc:1261] T 85593473500b4d5eaf9f8b0bf2296777 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:38.501257 22458 consensus_queue.cc:1035] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:38.510361 19636 consensus_queue.cc:237] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:38.512367 21267 raft_consensus.cc:1261] T db0f01ee72b24ec9b84410cf94ef9759 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:10:38.512660 19376 raft_consensus.cc:1261] T db0f01ee72b24ec9b84410cf94ef9759 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:10:38.513038 24190 consensus_queue.cc:1035] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:10:38.513151 21399 raft_consensus.cc:1261] T db0f01ee72b24ec9b84410cf94ef9759 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:10:38.513525 24190 consensus_queue.cc:1035] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:10:38.514935 24190 consensus_queue.cc:1035] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:10:38.516642 22458 raft_consensus.cc:2932] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:38.517172 21399 raft_consensus.cc:2932] T db0f01ee72b24ec9b84410cf94ef9759 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:38.519742 19232 catalog_manager.cc:5484] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 reported cstate change: config changed from index 5 to 6, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:38.520609 19376 raft_consensus.cc:2932] T db0f01ee72b24ec9b84410cf94ef9759 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:38.524287 19748 tablet_service.cc:1514] Processing DeleteTablet for tablet db0f01ee72b24ec9b84410cf94ef9759 with delete_type TABLET_DATA_TOMBSTONED (TS da3c4e948ff747678f18456c134d7ed4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36530
I20240501 14:10:38.526474 24749 tablet_replica.cc:330] T db0f01ee72b24ec9b84410cf94ef9759 P da3c4e948ff747678f18456c134d7ed4: stopping tablet replica
I20240501 14:10:38.526859 24749 raft_consensus.cc:2229] T db0f01ee72b24ec9b84410cf94ef9759 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:38.527490 24749 raft_consensus.cc:2258] T db0f01ee72b24ec9b84410cf94ef9759 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:38.530225 24749 ts_tablet_manager.cc:1875] T db0f01ee72b24ec9b84410cf94ef9759 P da3c4e948ff747678f18456c134d7ed4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:38.538826 24749 ts_tablet_manager.cc:1888] T db0f01ee72b24ec9b84410cf94ef9759 P da3c4e948ff747678f18456c134d7ed4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:10:38.538995 24749 log.cc:1196] T db0f01ee72b24ec9b84410cf94ef9759 P da3c4e948ff747678f18456c134d7ed4: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal/wals/db0f01ee72b24ec9b84410cf94ef9759
I20240501 14:10:38.548202 19220 catalog_manager.cc:4830] TS da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357): tablet db0f01ee72b24ec9b84410cf94ef9759 (table rebalance_test_table_2 [id=6f6b0a378a484e5889d733ff982d9eb7]) successfully deleted
I20240501 14:10:38.563704 24190 raft_consensus.cc:2932] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:38.565866 21266 raft_consensus.cc:2932] T 85593473500b4d5eaf9f8b0bf2296777 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:38.566525 19376 raft_consensus.cc:2932] T 85593473500b4d5eaf9f8b0bf2296777 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:38.569723 19232 catalog_manager.cc:5484] T 85593473500b4d5eaf9f8b0bf2296777 P e6246b183293412494919f1e7efc9448 reported cstate change: config changed from index 4 to 5, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:38.574642 21266 raft_consensus.cc:2932] T db0f01ee72b24ec9b84410cf94ef9759 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:38.574975 19748 tablet_service.cc:1514] Processing DeleteTablet for tablet 85593473500b4d5eaf9f8b0bf2296777 with delete_type TABLET_DATA_TOMBSTONED (TS da3c4e948ff747678f18456c134d7ed4 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:36530
I20240501 14:10:38.575345 24749 tablet_replica.cc:330] T 85593473500b4d5eaf9f8b0bf2296777 P da3c4e948ff747678f18456c134d7ed4: stopping tablet replica
I20240501 14:10:38.575507 24749 raft_consensus.cc:2229] T 85593473500b4d5eaf9f8b0bf2296777 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:38.575644 24749 raft_consensus.cc:2258] T 85593473500b4d5eaf9f8b0bf2296777 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:38.576234 24749 ts_tablet_manager.cc:1875] T 85593473500b4d5eaf9f8b0bf2296777 P da3c4e948ff747678f18456c134d7ed4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:38.578589 24749 ts_tablet_manager.cc:1888] T 85593473500b4d5eaf9f8b0bf2296777 P da3c4e948ff747678f18456c134d7ed4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:10:38.578662 24749 log.cc:1196] T 85593473500b4d5eaf9f8b0bf2296777 P da3c4e948ff747678f18456c134d7ed4: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal/wals/85593473500b4d5eaf9f8b0bf2296777
I20240501 14:10:38.580482 19220 catalog_manager.cc:4830] TS da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357): tablet 85593473500b4d5eaf9f8b0bf2296777 (table rebalance_test_table_0 [id=b52269ef7c494f74a5e4f0d5cb3b8a47]) successfully deleted
I20240501 14:10:38.636363 19376 consensus_queue.cc:237] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:38.637549 21136 raft_consensus.cc:1261] T f69020bb6c244848a6b0a50fd2b4f3eb P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:10:38.638031 23779 consensus_queue.cc:1035] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:10:38.639763 23779 raft_consensus.cc:2932] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:38.640089 21136 raft_consensus.cc:2932] T f69020bb6c244848a6b0a50fd2b4f3eb P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:38.642653 19233 catalog_manager.cc:5484] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc reported cstate change: config changed from index 5 to 6, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New cstate: current_term: 1 leader_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:38.645224 19485 tablet_service.cc:1514] Processing DeleteTablet for tablet f69020bb6c244848a6b0a50fd2b4f3eb with delete_type TABLET_DATA_TOMBSTONED (TS 99a0be6ef2144b7fb247ba9d3f051d66 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37600
I20240501 14:10:38.646270 19376 consensus_queue.cc:237] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
W20240501 14:10:38.649149 19309 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135:38751): Couldn't send request to peer 43ce4a4c954f4a5f80954b38933c189d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f69020bb6c244848a6b0a50fd2b4f3eb. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:38.649660 21136 raft_consensus.cc:1261] T f69020bb6c244848a6b0a50fd2b4f3eb P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:10:38.650087 23779 consensus_queue.cc:1035] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:10:38.650393 24842 tablet_replica.cc:330] T f69020bb6c244848a6b0a50fd2b4f3eb P 99a0be6ef2144b7fb247ba9d3f051d66: stopping tablet replica
I20240501 14:10:38.650815 24842 raft_consensus.cc:2229] T f69020bb6c244848a6b0a50fd2b4f3eb P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:38.651055 24842 raft_consensus.cc:2258] T f69020bb6c244848a6b0a50fd2b4f3eb P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:38.652319 23779 raft_consensus.cc:2932] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:38.652465 24842 ts_tablet_manager.cc:1875] T f69020bb6c244848a6b0a50fd2b4f3eb P 99a0be6ef2144b7fb247ba9d3f051d66: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:38.654379 19218 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet f69020bb6c244848a6b0a50fd2b4f3eb with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:10:38.655135 21136 raft_consensus.cc:2932] T f69020bb6c244848a6b0a50fd2b4f3eb P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:38.657186 24842 ts_tablet_manager.cc:1888] T f69020bb6c244848a6b0a50fd2b4f3eb P 99a0be6ef2144b7fb247ba9d3f051d66: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:10:38.657328 24842 log.cc:1196] T f69020bb6c244848a6b0a50fd2b4f3eb P 99a0be6ef2144b7fb247ba9d3f051d66: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal/wals/f69020bb6c244848a6b0a50fd2b4f3eb
I20240501 14:10:38.658380 19219 catalog_manager.cc:4830] TS 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829): tablet f69020bb6c244848a6b0a50fd2b4f3eb (table rebalance_test_table_4 [id=151dfad0ef40470ab9c5203c42400b60]) successfully deleted
I20240501 14:10:38.658711 19230 catalog_manager.cc:5484] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc reported cstate change: config changed from index 6 to 7, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New cstate: current_term: 1 leader_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:38.679044 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:38.693449 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:38.695392 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:38.697367 21247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:38.698062 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:38.698513 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:38.699112 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:38.711239 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:38.745512 19505 consensus_queue.cc:237] T b6be9a115fe14c098c29cad819849f95 P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:38.746960 21399 raft_consensus.cc:1261] T b6be9a115fe14c098c29cad819849f95 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Refusing update from remote peer 99a0be6ef2144b7fb247ba9d3f051d66: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:38.747138 19769 raft_consensus.cc:1261] T b6be9a115fe14c098c29cad819849f95 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Refusing update from remote peer 99a0be6ef2144b7fb247ba9d3f051d66: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:38.747673 23096 consensus_queue.cc:1035] T b6be9a115fe14c098c29cad819849f95 P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:38.748057 23096 consensus_queue.cc:1035] T b6be9a115fe14c098c29cad819849f95 P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:38.755390 24407 raft_consensus.cc:2932] T b6be9a115fe14c098c29cad819849f95 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:38.757517 21399 raft_consensus.cc:2932] T b6be9a115fe14c098c29cad819849f95 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:38.758086 19769 raft_consensus.cc:2932] T b6be9a115fe14c098c29cad819849f95 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:38.761495 19226 catalog_manager.cc:5484] T b6be9a115fe14c098c29cad819849f95 P 43ce4a4c954f4a5f80954b38933c189d reported cstate change: config changed from index 4 to 5, VOTER ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131) evicted. New cstate: current_term: 1 leader_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:38.767362 19617 tablet_service.cc:1514] Processing DeleteTablet for tablet b6be9a115fe14c098c29cad819849f95 with delete_type TABLET_DATA_TOMBSTONED (TS ec8cde17a00645e7b38d5d29f7425367 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:47238
I20240501 14:10:38.773699 24909 tablet_replica.cc:330] T b6be9a115fe14c098c29cad819849f95 P ec8cde17a00645e7b38d5d29f7425367: stopping tablet replica
I20240501 14:10:38.774592 24909 raft_consensus.cc:2229] T b6be9a115fe14c098c29cad819849f95 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:38.774840 24909 raft_consensus.cc:2258] T b6be9a115fe14c098c29cad819849f95 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:38.776396 24909 ts_tablet_manager.cc:1875] T b6be9a115fe14c098c29cad819849f95 P ec8cde17a00645e7b38d5d29f7425367: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:38.797510 24909 ts_tablet_manager.cc:1888] T b6be9a115fe14c098c29cad819849f95 P ec8cde17a00645e7b38d5d29f7425367: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:10:38.797716 24909 log.cc:1196] T b6be9a115fe14c098c29cad819849f95 P ec8cde17a00645e7b38d5d29f7425367: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal/wals/b6be9a115fe14c098c29cad819849f95
I20240501 14:10:38.798399 19219 catalog_manager.cc:4830] TS ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201): tablet b6be9a115fe14c098c29cad819849f95 (table rebalance_test_table_3 [id=35d6f8fac5394197b987aa44dab206c1]) successfully deleted
I20240501 14:10:38.821064 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:38.825974 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:38.831195 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:38.833496 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:38.848713 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:38.849023 21247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:38.854212 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:38.858315 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:38.881947 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:38.890690 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:38.898949 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:38.902987 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:38.904779 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:38.935889 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:38.936751 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:38.950242 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:38.950861 21247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:38.973657 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:38.974517 21247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:38.975507 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:38.992666 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:39.003938 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:39.011530 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:39.013146 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:39.065694 19505 consensus_queue.cc:237] T 55830424a98348e382759c54afdeabcd P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:10:39.070647 21522 raft_consensus.cc:1261] T 55830424a98348e382759c54afdeabcd P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Refusing update from remote peer 99a0be6ef2144b7fb247ba9d3f051d66: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:39.070943 19636 raft_consensus.cc:1261] T 55830424a98348e382759c54afdeabcd P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Refusing update from remote peer 99a0be6ef2144b7fb247ba9d3f051d66: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:39.071389 24407 consensus_queue.cc:1035] T 55830424a98348e382759c54afdeabcd P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:39.073478 24407 consensus_queue.cc:1035] T 55830424a98348e382759c54afdeabcd P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:39.078745 24407 raft_consensus.cc:2932] T 55830424a98348e382759c54afdeabcd P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:39.081518 19226 catalog_manager.cc:5484] T 55830424a98348e382759c54afdeabcd P 99a0be6ef2144b7fb247ba9d3f051d66 reported cstate change: config changed from index 4 to 5, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) evicted. New cstate: current_term: 1 leader_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:39.081754 19636 raft_consensus.cc:2932] T 55830424a98348e382759c54afdeabcd P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:39.089848 19356 tablet_service.cc:1514] Processing DeleteTablet for tablet 55830424a98348e382759c54afdeabcd with delete_type TABLET_DATA_TOMBSTONED (TS fb1e6bc2d07b4ce3b622a66f14c93bdc not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:46838
I20240501 14:10:39.096972 25055 tablet_replica.cc:330] T 55830424a98348e382759c54afdeabcd P fb1e6bc2d07b4ce3b622a66f14c93bdc: stopping tablet replica
I20240501 14:10:39.097007 19769 consensus_queue.cc:237] T cf62635e9a994259b9e293e5eb425b55 P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:10:39.097293 25055 raft_consensus.cc:2229] T 55830424a98348e382759c54afdeabcd P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:39.097508 25055 raft_consensus.cc:2258] T 55830424a98348e382759c54afdeabcd P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:39.098132 25055 ts_tablet_manager.cc:1875] T 55830424a98348e382759c54afdeabcd P fb1e6bc2d07b4ce3b622a66f14c93bdc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:39.098997 21530 raft_consensus.cc:1261] T cf62635e9a994259b9e293e5eb425b55 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:39.099496 19636 raft_consensus.cc:1261] T cf62635e9a994259b9e293e5eb425b55 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:39.100803 25055 ts_tablet_manager.cc:1888] T 55830424a98348e382759c54afdeabcd P fb1e6bc2d07b4ce3b622a66f14c93bdc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:10:39.101543 24191 consensus_queue.cc:1035] T cf62635e9a994259b9e293e5eb425b55 P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:39.101965 24191 consensus_queue.cc:1035] T cf62635e9a994259b9e293e5eb425b55 P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:39.104769 21522 raft_consensus.cc:2932] T 55830424a98348e382759c54afdeabcd P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:39.107106 24191 raft_consensus.cc:2932] T cf62635e9a994259b9e293e5eb425b55 P da3c4e948ff747678f18456c134d7ed4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:39.109251 21530 raft_consensus.cc:2932] T cf62635e9a994259b9e293e5eb425b55 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:39.111640 19231 catalog_manager.cc:5484] T cf62635e9a994259b9e293e5eb425b55 P da3c4e948ff747678f18456c134d7ed4 reported cstate change: config changed from index 4 to 5, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New cstate: current_term: 1 leader_uuid: "da3c4e948ff747678f18456c134d7ed4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:39.112354 25055 log.cc:1196] T 55830424a98348e382759c54afdeabcd P fb1e6bc2d07b4ce3b622a66f14c93bdc: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal/wals/55830424a98348e382759c54afdeabcd
I20240501 14:10:39.114030 19218 catalog_manager.cc:4830] TS fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847): tablet 55830424a98348e382759c54afdeabcd (table rebalance_test_table_0 [id=b52269ef7c494f74a5e4f0d5cb3b8a47]) successfully deleted
I20240501 14:10:39.117583 19485 tablet_service.cc:1514] Processing DeleteTablet for tablet cf62635e9a994259b9e293e5eb425b55 with delete_type TABLET_DATA_TOMBSTONED (TS 99a0be6ef2144b7fb247ba9d3f051d66 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:37600
I20240501 14:10:39.117805 24842 tablet_replica.cc:330] T cf62635e9a994259b9e293e5eb425b55 P 99a0be6ef2144b7fb247ba9d3f051d66: stopping tablet replica
I20240501 14:10:39.117930 24842 raft_consensus.cc:2229] T cf62635e9a994259b9e293e5eb425b55 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:39.118046 24842 raft_consensus.cc:2258] T cf62635e9a994259b9e293e5eb425b55 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:39.118585 24842 ts_tablet_manager.cc:1875] T cf62635e9a994259b9e293e5eb425b55 P 99a0be6ef2144b7fb247ba9d3f051d66: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:39.120978 24842 ts_tablet_manager.cc:1888] T cf62635e9a994259b9e293e5eb425b55 P 99a0be6ef2144b7fb247ba9d3f051d66: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:10:39.121059 24842 log.cc:1196] T cf62635e9a994259b9e293e5eb425b55 P 99a0be6ef2144b7fb247ba9d3f051d66: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal/wals/cf62635e9a994259b9e293e5eb425b55
I20240501 14:10:39.121114 19636 raft_consensus.cc:2932] T cf62635e9a994259b9e293e5eb425b55 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:39.122933 19219 catalog_manager.cc:4830] TS 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829): tablet cf62635e9a994259b9e293e5eb425b55 (table rebalance_test_table_4 [id=151dfad0ef40470ab9c5203c42400b60]) successfully deleted
I20240501 14:10:39.166150 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:39.186280 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:39.206308 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:39.222705 21247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:39.234308 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:39.241014 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:39.266443 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:39.277432 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:39.277734 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:39.281759 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:39.283460 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:39.304895 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:39.317729 21247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:39.328908 25099 ts_tablet_manager.cc:902] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d: Initiating tablet copy from peer fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847)
I20240501 14:10:39.329447 25099 tablet_copy_client.cc:308] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.129:35847
I20240501 14:10:39.329872 19396 tablet_copy_service.cc:140] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Received BeginTabletCopySession request for tablet f69020bb6c244848a6b0a50fd2b4f3eb from peer 43ce4a4c954f4a5f80954b38933c189d ({username='slave'} at 127.7.39.135:49533)
I20240501 14:10:39.329970 19396 tablet_copy_service.cc:161] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Beginning new tablet copy session on tablet f69020bb6c244848a6b0a50fd2b4f3eb from peer 43ce4a4c954f4a5f80954b38933c189d at {username='slave'} at 127.7.39.135:49533: session id = 43ce4a4c954f4a5f80954b38933c189d-f69020bb6c244848a6b0a50fd2b4f3eb
I20240501 14:10:39.330583 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:39.330914 19396 tablet_copy_source_session.cc:205] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:39.332114 25099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f69020bb6c244848a6b0a50fd2b4f3eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:39.336019 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:39.359325 19769 consensus_queue.cc:237] T 7eea58d4a33f422a9ada1e82d408577f P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:39.361143 21399 raft_consensus.cc:1261] T 7eea58d4a33f422a9ada1e82d408577f P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:10:39.362848 21136 raft_consensus.cc:1261] T 7eea58d4a33f422a9ada1e82d408577f P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:10:39.363924 24191 consensus_queue.cc:1035] T 7eea58d4a33f422a9ada1e82d408577f P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:10:39.365290 24191 consensus_queue.cc:1035] T 7eea58d4a33f422a9ada1e82d408577f P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:10:39.367305 19505 raft_consensus.cc:1261] T 7eea58d4a33f422a9ada1e82d408577f P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:10:39.368691 24191 consensus_queue.cc:1035] T 7eea58d4a33f422a9ada1e82d408577f P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:10:39.377151 24191 raft_consensus.cc:2932] T 7eea58d4a33f422a9ada1e82d408577f P da3c4e948ff747678f18456c134d7ed4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:39.380060 21136 raft_consensus.cc:2932] T 7eea58d4a33f422a9ada1e82d408577f P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:39.380427 25099 tablet_copy_client.cc:783] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:39.380662 25099 tablet_copy_client.cc:647] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:39.381961 25099 tablet_copy_client.cc:523] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:39.382691 21399 raft_consensus.cc:2932] T 7eea58d4a33f422a9ada1e82d408577f P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:39.383080 19228 catalog_manager.cc:5484] T 7eea58d4a33f422a9ada1e82d408577f P da3c4e948ff747678f18456c134d7ed4 reported cstate change: config changed from index 5 to 6, VOTER ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131) evicted. New cstate: current_term: 1 leader_uuid: "da3c4e948ff747678f18456c134d7ed4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:39.386332 19617 tablet_service.cc:1514] Processing DeleteTablet for tablet 7eea58d4a33f422a9ada1e82d408577f with delete_type TABLET_DATA_TOMBSTONED (TS ec8cde17a00645e7b38d5d29f7425367 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47238
I20240501 14:10:39.390961 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:39.392292 19505 raft_consensus.cc:2932] T 7eea58d4a33f422a9ada1e82d408577f P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:39.392704 25099 tablet_bootstrap.cc:492] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d: Bootstrap starting.
I20240501 14:10:39.394827 25161 tablet_replica.cc:330] T 7eea58d4a33f422a9ada1e82d408577f P ec8cde17a00645e7b38d5d29f7425367: stopping tablet replica
I20240501 14:10:39.395030 25161 raft_consensus.cc:2229] T 7eea58d4a33f422a9ada1e82d408577f P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:39.395222 25161 raft_consensus.cc:2258] T 7eea58d4a33f422a9ada1e82d408577f P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:39.395676 25161 ts_tablet_manager.cc:1875] T 7eea58d4a33f422a9ada1e82d408577f P ec8cde17a00645e7b38d5d29f7425367: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:39.397982 25161 ts_tablet_manager.cc:1888] T 7eea58d4a33f422a9ada1e82d408577f P ec8cde17a00645e7b38d5d29f7425367: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:10:39.398118 25161 log.cc:1196] T 7eea58d4a33f422a9ada1e82d408577f P ec8cde17a00645e7b38d5d29f7425367: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal/wals/7eea58d4a33f422a9ada1e82d408577f
I20240501 14:10:39.398638 19219 catalog_manager.cc:4830] TS ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201): tablet 7eea58d4a33f422a9ada1e82d408577f (table rebalance_test_table_9 [id=1fbbd460d13647c1a85b517926267158]) successfully deleted
I20240501 14:10:39.401495 19376 consensus_queue.cc:237] T 0988c468757d49738765bdcb31794bd1 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:39.403448 19769 raft_consensus.cc:1261] T 0988c468757d49738765bdcb31794bd1 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:39.403862 24838 consensus_queue.cc:1035] T 0988c468757d49738765bdcb31794bd1 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:39.404453 21136 raft_consensus.cc:1261] T 0988c468757d49738765bdcb31794bd1 P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:39.405082 24838 consensus_queue.cc:1035] T 0988c468757d49738765bdcb31794bd1 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:39.413652 24838 raft_consensus.cc:2932] T 0988c468757d49738765bdcb31794bd1 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:39.418375 19228 catalog_manager.cc:5484] T 0988c468757d49738765bdcb31794bd1 P fb1e6bc2d07b4ce3b622a66f14c93bdc reported cstate change: config changed from index 4 to 5, VOTER ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131) evicted. New cstate: current_term: 1 leader_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:39.422068 19769 raft_consensus.cc:2932] T 0988c468757d49738765bdcb31794bd1 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:39.425061 19617 tablet_service.cc:1514] Processing DeleteTablet for tablet 0988c468757d49738765bdcb31794bd1 with delete_type TABLET_DATA_TOMBSTONED (TS ec8cde17a00645e7b38d5d29f7425367 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:47238
I20240501 14:10:39.425354 25161 tablet_replica.cc:330] T 0988c468757d49738765bdcb31794bd1 P ec8cde17a00645e7b38d5d29f7425367: stopping tablet replica
I20240501 14:10:39.425509 25161 raft_consensus.cc:2229] T 0988c468757d49738765bdcb31794bd1 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:39.425664 25161 raft_consensus.cc:2258] T 0988c468757d49738765bdcb31794bd1 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:39.426331 25161 ts_tablet_manager.cc:1875] T 0988c468757d49738765bdcb31794bd1 P ec8cde17a00645e7b38d5d29f7425367: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:39.428741 25161 ts_tablet_manager.cc:1888] T 0988c468757d49738765bdcb31794bd1 P ec8cde17a00645e7b38d5d29f7425367: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:10:39.428814 25161 log.cc:1196] T 0988c468757d49738765bdcb31794bd1 P ec8cde17a00645e7b38d5d29f7425367: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal/wals/0988c468757d49738765bdcb31794bd1
I20240501 14:10:39.429507 19219 catalog_manager.cc:4830] TS ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201): tablet 0988c468757d49738765bdcb31794bd1 (table rebalance_test_table_7 [id=ed1a7ed87fa9480ba18a052130ecdc05]) successfully deleted
I20240501 14:10:39.432170 21136 raft_consensus.cc:2932] T 0988c468757d49738765bdcb31794bd1 P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:39.436497 25099 tablet_bootstrap.cc:492] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:39.437611 25099 tablet_bootstrap.cc:492] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d: Bootstrap complete.
I20240501 14:10:39.437866 25099 ts_tablet_manager.cc:1367] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d: Time spent bootstrapping tablet: real 0.045s user 0.028s sys 0.001s
I20240501 14:10:39.439851 25099 raft_consensus.cc:348] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:39.440013 25099 raft_consensus.cc:729] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43ce4a4c954f4a5f80954b38933c189d, State: Initialized, Role: FOLLOWER
I20240501 14:10:39.440186 25099 consensus_queue.cc:260] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:39.441419 25099 ts_tablet_manager.cc:1398] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:10:39.442126 19396 tablet_copy_service.cc:342] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Request end of tablet copy session 43ce4a4c954f4a5f80954b38933c189d-f69020bb6c244848a6b0a50fd2b4f3eb received from {username='slave'} at 127.7.39.135:49533
I20240501 14:10:39.442209 19396 tablet_copy_service.cc:433] P fb1e6bc2d07b4ce3b622a66f14c93bdc: ending tablet copy session 43ce4a4c954f4a5f80954b38933c189d-f69020bb6c244848a6b0a50fd2b4f3eb on tablet f69020bb6c244848a6b0a50fd2b4f3eb with peer 43ce4a4c954f4a5f80954b38933c189d
I20240501 14:10:39.484102 19636 consensus_queue.cc:237] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:39.486102 21266 raft_consensus.cc:1261] T 2dcca6e0e6374520bd4522c80c110e97 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:10:39.486480 21399 raft_consensus.cc:1261] T 2dcca6e0e6374520bd4522c80c110e97 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:10:39.486896 19505 raft_consensus.cc:1261] T 2dcca6e0e6374520bd4522c80c110e97 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:10:39.487953 24775 consensus_queue.cc:1035] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:10:39.488286 25110 consensus_queue.cc:1035] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:10:39.492506 24775 consensus_queue.cc:1035] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:10:39.498239 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:39.504251 24190 raft_consensus.cc:2932] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:39.506206 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:39.507576 21399 raft_consensus.cc:2932] T 2dcca6e0e6374520bd4522c80c110e97 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:39.510722 19228 catalog_manager.cc:5484] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 reported cstate change: config changed from index 5 to 6, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) evicted. New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:39.513319 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:39.514034 21266 raft_consensus.cc:2932] T 2dcca6e0e6374520bd4522c80c110e97 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:39.514582 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:39.517144 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:39.519448 19356 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dcca6e0e6374520bd4522c80c110e97 with delete_type TABLET_DATA_TOMBSTONED (TS fb1e6bc2d07b4ce3b622a66f14c93bdc not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46838
I20240501 14:10:39.519670 25055 tablet_replica.cc:330] T 2dcca6e0e6374520bd4522c80c110e97 P fb1e6bc2d07b4ce3b622a66f14c93bdc: stopping tablet replica
I20240501 14:10:39.519814 25055 raft_consensus.cc:2229] T 2dcca6e0e6374520bd4522c80c110e97 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:39.519990 25055 raft_consensus.cc:2258] T 2dcca6e0e6374520bd4522c80c110e97 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:39.520560 25055 ts_tablet_manager.cc:1875] T 2dcca6e0e6374520bd4522c80c110e97 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:39.522500 25055 ts_tablet_manager.cc:1888] T 2dcca6e0e6374520bd4522c80c110e97 P fb1e6bc2d07b4ce3b622a66f14c93bdc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:10:39.522552 25055 log.cc:1196] T 2dcca6e0e6374520bd4522c80c110e97 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal/wals/2dcca6e0e6374520bd4522c80c110e97
I20240501 14:10:39.523010 19218 catalog_manager.cc:4830] TS fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847): tablet 2dcca6e0e6374520bd4522c80c110e97 (table rebalance_test_table_8 [id=3866c35dd0e64d87b6049e5e14cb0d31]) successfully deleted
I20240501 14:10:39.525599 21247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:39.535403 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:39.539412 21399 raft_consensus.cc:1203] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:10:39.540467 19505 raft_consensus.cc:2932] T 2dcca6e0e6374520bd4522c80c110e97 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:39.550410 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:39.589192 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:39.595758 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:39.613255 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:39.617946 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:39.617956 21247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:39.634806 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:39.635275 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:39.644899 19375 consensus_queue.cc:237] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:39.646410 19506 raft_consensus.cc:1261] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:39.646621 21136 raft_consensus.cc:1261] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:39.646833 25181 consensus_queue.cc:1035] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:39.646889 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:39.647310 25181 consensus_queue.cc:1035] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:39.660882 24847 raft_consensus.cc:2932] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:39.661265 19506 raft_consensus.cc:2932] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:39.661706 21136 raft_consensus.cc:2932] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:39.663836 19225 catalog_manager.cc:5484] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P fb1e6bc2d07b4ce3b622a66f14c93bdc reported cstate change: config changed from index 4 to 5, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New cstate: current_term: 1 leader_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:39.667269 19375 consensus_queue.cc:237] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:39.668525 19748 tablet_service.cc:1514] Processing DeleteTablet for tablet 78e14dea8b6e4c0abd1c88dcd5b4e608 with delete_type TABLET_DATA_TOMBSTONED (TS da3c4e948ff747678f18456c134d7ed4 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:36530
I20240501 14:10:39.668948 19767 raft_consensus.cc:1261] T ad7c255baee74e629e2976a942a4174f P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:39.669121 21399 raft_consensus.cc:1261] T ad7c255baee74e629e2976a942a4174f P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:39.669471 24847 consensus_queue.cc:1035] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:39.669613 25181 consensus_queue.cc:1035] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:39.671319 24847 raft_consensus.cc:2932] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:39.673054 21399 raft_consensus.cc:2932] T ad7c255baee74e629e2976a942a4174f P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:39.675542 19226 catalog_manager.cc:5484] T ad7c255baee74e629e2976a942a4174f P 43ce4a4c954f4a5f80954b38933c189d reported cstate change: config changed from index 4 to 5, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New cstate: current_term: 1 leader_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:39.679602 19485 tablet_service.cc:1514] Processing DeleteTablet for tablet ad7c255baee74e629e2976a942a4174f with delete_type TABLET_DATA_TOMBSTONED (TS 99a0be6ef2144b7fb247ba9d3f051d66 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:37600
I20240501 14:10:39.688375 25328 tablet_replica.cc:330] T ad7c255baee74e629e2976a942a4174f P 99a0be6ef2144b7fb247ba9d3f051d66: stopping tablet replica
I20240501 14:10:39.688561 25328 raft_consensus.cc:2229] T ad7c255baee74e629e2976a942a4174f P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:39.688794 25328 raft_consensus.cc:2258] T ad7c255baee74e629e2976a942a4174f P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:39.689397 25328 ts_tablet_manager.cc:1875] T ad7c255baee74e629e2976a942a4174f P 99a0be6ef2144b7fb247ba9d3f051d66: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:39.691910 25328 ts_tablet_manager.cc:1888] T ad7c255baee74e629e2976a942a4174f P 99a0be6ef2144b7fb247ba9d3f051d66: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:10:39.692175 25321 tablet_replica.cc:330] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P da3c4e948ff747678f18456c134d7ed4: stopping tablet replica
I20240501 14:10:39.692317 25321 raft_consensus.cc:2229] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:39.692487 25321 raft_consensus.cc:2258] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:39.692941 25321 ts_tablet_manager.cc:1875] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P da3c4e948ff747678f18456c134d7ed4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:39.694665 25328 log.cc:1196] T ad7c255baee74e629e2976a942a4174f P 99a0be6ef2144b7fb247ba9d3f051d66: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal/wals/ad7c255baee74e629e2976a942a4174f
I20240501 14:10:39.696324 19219 catalog_manager.cc:4830] TS 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829): tablet ad7c255baee74e629e2976a942a4174f (table rebalance_test_table_6 [id=afc36d383e3e4795ad11804e9bebcb51]) successfully deleted
I20240501 14:10:39.696460 25321 ts_tablet_manager.cc:1888] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P da3c4e948ff747678f18456c134d7ed4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:10:39.696589 25321 log.cc:1196] T 78e14dea8b6e4c0abd1c88dcd5b4e608 P da3c4e948ff747678f18456c134d7ed4: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal/wals/78e14dea8b6e4c0abd1c88dcd5b4e608
I20240501 14:10:39.696954 19767 raft_consensus.cc:2932] T ad7c255baee74e629e2976a942a4174f P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:39.698928 19220 catalog_manager.cc:4830] TS da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357): tablet 78e14dea8b6e4c0abd1c88dcd5b4e608 (table rebalance_test_table_2 [id=6f6b0a378a484e5889d733ff982d9eb7]) successfully deleted
I20240501 14:10:39.776603 21247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:39.778582 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:39.779492 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:39.790990 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:39.793291 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:39.796857 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:39.798128 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:39.806265 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:39.889524 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:39.891355 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:39.894235 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:39.897336 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:39.907502 21247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:39.910900 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:39.939846 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:39.961377 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:39.965479 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:39.977880 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:40.000907 21247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:40.002666 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:40.010859 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:40.012157 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:40.012339 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:40.014084 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:40.028932 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:40.042878 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:40.047923 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:40.062114 21247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:40.062954 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:40.071436 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:40.084017 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:40.092537 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:40.145359 19637 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "12ffa74f7a5b472aac57253693273cad"
dest_uuid: "ec8cde17a00645e7b38d5d29f7425367"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42376
I20240501 14:10:40.145565 19637 raft_consensus.cc:595] T 12ffa74f7a5b472aac57253693273cad P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:10:40.161406 19767 consensus_queue.cc:237] T bb291a0c743c4e638f79da36cc77faa3 P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:40.163584 21137 raft_consensus.cc:1261] T bb291a0c743c4e638f79da36cc77faa3 P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:10:40.163688 21530 raft_consensus.cc:1261] T bb291a0c743c4e638f79da36cc77faa3 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:10:40.164335 25071 consensus_queue.cc:1035] T bb291a0c743c4e638f79da36cc77faa3 P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:10:40.164719 25071 consensus_queue.cc:1035] T bb291a0c743c4e638f79da36cc77faa3 P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:10:40.165442 19375 raft_consensus.cc:1261] T bb291a0c743c4e638f79da36cc77faa3 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:10:40.166115 25071 consensus_queue.cc:1035] T bb291a0c743c4e638f79da36cc77faa3 P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:10:40.178834 25603 raft_consensus.cc:2932] T bb291a0c743c4e638f79da36cc77faa3 P da3c4e948ff747678f18456c134d7ed4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:40.181620 21530 raft_consensus.cc:2932] T bb291a0c743c4e638f79da36cc77faa3 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:40.183200 19506 consensus_queue.cc:237] T f6ca088ce3aa4b97aa54c8ce6d700921 P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:40.183341 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:40.184712 21399 raft_consensus.cc:1261] T f6ca088ce3aa4b97aa54c8ce6d700921 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Refusing update from remote peer 99a0be6ef2144b7fb247ba9d3f051d66: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:40.184772 19767 raft_consensus.cc:1261] T f6ca088ce3aa4b97aa54c8ce6d700921 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Refusing update from remote peer 99a0be6ef2144b7fb247ba9d3f051d66: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:40.185137 24892 consensus_queue.cc:1035] T f6ca088ce3aa4b97aa54c8ce6d700921 P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:40.185453 24892 consensus_queue.cc:1035] T f6ca088ce3aa4b97aa54c8ce6d700921 P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:40.188446 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:40.192752 21137 raft_consensus.cc:2932] T bb291a0c743c4e638f79da36cc77faa3 P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:40.193013 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:40.196878 19228 catalog_manager.cc:5484] T bb291a0c743c4e638f79da36cc77faa3 P da3c4e948ff747678f18456c134d7ed4 reported cstate change: config changed from index 5 to 6, VOTER ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131) evicted. New cstate: current_term: 1 leader_uuid: "da3c4e948ff747678f18456c134d7ed4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:40.205499 21247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:40.206636 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:40.211851 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:40.218437 19375 raft_consensus.cc:2932] T bb291a0c743c4e638f79da36cc77faa3 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:40.225306 19375 consensus_queue.cc:237] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:40.227444 21398 raft_consensus.cc:1261] T ad7c255baee74e629e2976a942a4174f P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:10:40.227852 24838 consensus_queue.cc:1035] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:10:40.230404 24838 raft_consensus.cc:2932] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:40.233004 21398 raft_consensus.cc:2932] T ad7c255baee74e629e2976a942a4174f P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:40.235019 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:40.238250 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:40.243124 24892 raft_consensus.cc:2932] T f6ca088ce3aa4b97aa54c8ce6d700921 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:40.243649 19767 raft_consensus.cc:2932] T f6ca088ce3aa4b97aa54c8ce6d700921 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:40.244585 24190 raft_consensus.cc:979] T 12ffa74f7a5b472aac57253693273cad P ec8cde17a00645e7b38d5d29f7425367: : Instructing follower e6246b183293412494919f1e7efc9448 to start an election
I20240501 14:10:40.244676 24190 raft_consensus.cc:1067] T 12ffa74f7a5b472aac57253693273cad P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Signalling peer e6246b183293412494919f1e7efc9448 to start an election
I20240501 14:10:40.245349 21268 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "12ffa74f7a5b472aac57253693273cad"
dest_uuid: "e6246b183293412494919f1e7efc9448"
from {username='slave'} at 127.7.39.131:43999
I20240501 14:10:40.245541 21268 raft_consensus.cc:482] T 12ffa74f7a5b472aac57253693273cad P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:10:40.245416 21399 raft_consensus.cc:2932] T f6ca088ce3aa4b97aa54c8ce6d700921 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:40.245643 21268 raft_consensus.cc:3037] T 12ffa74f7a5b472aac57253693273cad P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:40.246763 21268 raft_consensus.cc:504] T 12ffa74f7a5b472aac57253693273cad P e6246b183293412494919f1e7efc9448 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:40.247790 19226 catalog_manager.cc:5484] T f6ca088ce3aa4b97aa54c8ce6d700921 P 99a0be6ef2144b7fb247ba9d3f051d66 reported cstate change: config changed from index 4 to 5, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) evicted. New cstate: current_term: 1 leader_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:40.248808 19637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12ffa74f7a5b472aac57253693273cad" candidate_uuid: "e6246b183293412494919f1e7efc9448" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ec8cde17a00645e7b38d5d29f7425367"
I20240501 14:10:40.248927 19637 raft_consensus.cc:3032] T 12ffa74f7a5b472aac57253693273cad P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:10:40.248963 19637 raft_consensus.cc:729] T 12ffa74f7a5b472aac57253693273cad P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ec8cde17a00645e7b38d5d29f7425367, State: Running, Role: LEADER
I20240501 14:10:40.249109 19637 consensus_queue.cc:260] T 12ffa74f7a5b472aac57253693273cad P ec8cde17a00645e7b38d5d29f7425367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:40.249267 19637 raft_consensus.cc:3037] T 12ffa74f7a5b472aac57253693273cad P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:40.250267 19637 raft_consensus.cc:2454] T 12ffa74f7a5b472aac57253693273cad P ec8cde17a00645e7b38d5d29f7425367 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6246b183293412494919f1e7efc9448 in term 2.
I20240501 14:10:40.253261 21268 leader_election.cc:288] T 12ffa74f7a5b472aac57253693273cad P e6246b183293412494919f1e7efc9448 [CANDIDATE]: Term 2 election: Requested vote from peers fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847), ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201), da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357)
I20240501 14:10:40.253695 19767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12ffa74f7a5b472aac57253693273cad" candidate_uuid: "e6246b183293412494919f1e7efc9448" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "da3c4e948ff747678f18456c134d7ed4"
I20240501 14:10:40.253829 19767 raft_consensus.cc:3037] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:40.254776 19767 raft_consensus.cc:2454] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6246b183293412494919f1e7efc9448 in term 2.
I20240501 14:10:40.255782 21203 leader_election.cc:302] T 12ffa74f7a5b472aac57253693273cad P e6246b183293412494919f1e7efc9448 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: da3c4e948ff747678f18456c134d7ed4, e6246b183293412494919f1e7efc9448, ec8cde17a00645e7b38d5d29f7425367; no voters:
I20240501 14:10:40.258461 25654 raft_consensus.cc:2781] T 12ffa74f7a5b472aac57253693273cad P e6246b183293412494919f1e7efc9448 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:10:40.258626 25654 raft_consensus.cc:686] T 12ffa74f7a5b472aac57253693273cad P e6246b183293412494919f1e7efc9448 [term 2 LEADER]: Becoming Leader. State: Replica: e6246b183293412494919f1e7efc9448, State: Running, Role: LEADER
I20240501 14:10:40.258944 25654 consensus_queue.cc:237] T 12ffa74f7a5b472aac57253693273cad P e6246b183293412494919f1e7efc9448 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:40.259146 19234 catalog_manager.cc:5484] T ad7c255baee74e629e2976a942a4174f P 43ce4a4c954f4a5f80954b38933c189d reported cstate change: config changed from index 5 to 6, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New cstate: current_term: 1 leader_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:40.264366 19225 catalog_manager.cc:5484] T 12ffa74f7a5b472aac57253693273cad P e6246b183293412494919f1e7efc9448 reported cstate change: term changed from 1 to 2, leader changed from ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131) to e6246b183293412494919f1e7efc9448 (127.7.39.134). New cstate: current_term: 2 leader_uuid: "e6246b183293412494919f1e7efc9448" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:40.267735 19375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12ffa74f7a5b472aac57253693273cad" candidate_uuid: "e6246b183293412494919f1e7efc9448" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc"
I20240501 14:10:40.267875 19375 raft_consensus.cc:3037] T 12ffa74f7a5b472aac57253693273cad P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:40.268918 19375 raft_consensus.cc:2454] T 12ffa74f7a5b472aac57253693273cad P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6246b183293412494919f1e7efc9448 in term 2.
I20240501 14:10:40.280875 19617 tablet_service.cc:1514] Processing DeleteTablet for tablet bb291a0c743c4e638f79da36cc77faa3 with delete_type TABLET_DATA_TOMBSTONED (TS ec8cde17a00645e7b38d5d29f7425367 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47238
I20240501 14:10:40.285809 19748 tablet_service.cc:1514] Processing DeleteTablet for tablet ad7c255baee74e629e2976a942a4174f with delete_type TABLET_DATA_TOMBSTONED (TS da3c4e948ff747678f18456c134d7ed4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36530
I20240501 14:10:40.287052 19375 consensus_queue.cc:237] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:40.288043 25671 tablet_replica.cc:330] T bb291a0c743c4e638f79da36cc77faa3 P ec8cde17a00645e7b38d5d29f7425367: stopping tablet replica
I20240501 14:10:40.288235 25671 raft_consensus.cc:2229] T bb291a0c743c4e638f79da36cc77faa3 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:40.288476 19356 tablet_service.cc:1514] Processing DeleteTablet for tablet f6ca088ce3aa4b97aa54c8ce6d700921 with delete_type TABLET_DATA_TOMBSTONED (TS fb1e6bc2d07b4ce3b622a66f14c93bdc not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:46838
I20240501 14:10:40.288821 25673 tablet_replica.cc:330] T f6ca088ce3aa4b97aa54c8ce6d700921 P fb1e6bc2d07b4ce3b622a66f14c93bdc: stopping tablet replica
I20240501 14:10:40.289005 25673 raft_consensus.cc:2229] T f6ca088ce3aa4b97aa54c8ce6d700921 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:40.289183 25673 raft_consensus.cc:2258] T f6ca088ce3aa4b97aa54c8ce6d700921 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:40.289875 25671 raft_consensus.cc:2258] T bb291a0c743c4e638f79da36cc77faa3 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:40.289949 25673 ts_tablet_manager.cc:1875] T f6ca088ce3aa4b97aa54c8ce6d700921 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:40.290493 25671 ts_tablet_manager.cc:1875] T bb291a0c743c4e638f79da36cc77faa3 P ec8cde17a00645e7b38d5d29f7425367: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:40.291040 21399 raft_consensus.cc:1261] T ad7c255baee74e629e2976a942a4174f P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:10:40.291649 25641 consensus_queue.cc:1035] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:10:40.292140 19767 raft_consensus.cc:1261] T ad7c255baee74e629e2976a942a4174f P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:10:40.292599 25673 ts_tablet_manager.cc:1888] T f6ca088ce3aa4b97aa54c8ce6d700921 P fb1e6bc2d07b4ce3b622a66f14c93bdc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:10:40.292743 25673 log.cc:1196] T f6ca088ce3aa4b97aa54c8ce6d700921 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal/wals/f6ca088ce3aa4b97aa54c8ce6d700921
I20240501 14:10:40.292690 25641 consensus_queue.cc:1035] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:10:40.294411 19767 raft_consensus.cc:858] T ad7c255baee74e629e2976a942a4174f P da3c4e948ff747678f18456c134d7ed4 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }.
Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }:
New config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
W20240501 14:10:40.294478 19767 raft_consensus.cc:1565] T ad7c255baee74e629e2976a942a4174f P da3c4e948ff747678f18456c134d7ed4 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 7' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }.
Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }:
New config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:40.294785 19767 raft_consensus.cc:2932] T ad7c255baee74e629e2976a942a4174f P da3c4e948ff747678f18456c134d7ed4 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:40.295320 25641 raft_consensus.cc:2932] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } }
I20240501 14:10:40.298003 19218 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet ad7c255baee74e629e2976a942a4174f with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:10:40.299595 19218 catalog_manager.cc:4830] TS fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847): tablet f6ca088ce3aa4b97aa54c8ce6d700921 (table rebalance_test_table_8 [id=3866c35dd0e64d87b6049e5e14cb0d31]) successfully deleted
I20240501 14:10:40.300315 19767 raft_consensus.cc:1467] T ad7c255baee74e629e2976a942a4174f P da3c4e948ff747678f18456c134d7ed4 [term 1 NON_PARTICIPANT]: Allowing update even though not a member of the config
I20240501 14:10:40.300490 21399 raft_consensus.cc:2932] T ad7c255baee74e629e2976a942a4174f P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } }
I20240501 14:10:40.300659 19767 raft_consensus.cc:2932] T ad7c255baee74e629e2976a942a4174f P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } }
I20240501 14:10:40.302256 25671 ts_tablet_manager.cc:1888] T bb291a0c743c4e638f79da36cc77faa3 P ec8cde17a00645e7b38d5d29f7425367: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:10:40.302373 25671 log.cc:1196] T bb291a0c743c4e638f79da36cc77faa3 P ec8cde17a00645e7b38d5d29f7425367: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal/wals/bb291a0c743c4e638f79da36cc77faa3
I20240501 14:10:40.303004 19219 catalog_manager.cc:4830] TS ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201): tablet bb291a0c743c4e638f79da36cc77faa3 (table rebalance_test_table_5 [id=4d7621338643426d90fe03eb71654cb8]) successfully deleted
I20240501 14:10:40.303757 19225 catalog_manager.cc:5484] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc reported cstate change: config changed from index 6 to 7, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) added. New cstate: current_term: 1 leader_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } }
W20240501 14:10:40.306439 19220 catalog_manager.cc:4804] TS da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357): delete failed for tablet ad7c255baee74e629e2976a942a4174f because of a CAS failure. No further retry: Illegal state: Request specified cas_config_opid_index_less_or_equal of 5 but the committed config has opid_index of 7
I20240501 14:10:40.336467 19375 consensus_queue.cc:237] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:40.338136 21268 raft_consensus.cc:1261] T d63afd081986473aa2715d074b123a93 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:40.338487 19767 raft_consensus.cc:1261] T d63afd081986473aa2715d074b123a93 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:40.338986 24838 consensus_queue.cc:1035] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:40.340226 24838 consensus_queue.cc:1035] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:40.368065 24838 raft_consensus.cc:2932] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:40.368758 19767 raft_consensus.cc:2932] T d63afd081986473aa2715d074b123a93 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:40.371158 19231 catalog_manager.cc:5484] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc reported cstate change: config changed from index 4 to 5, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New cstate: current_term: 1 leader_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:40.372864 21268 raft_consensus.cc:2932] T d63afd081986473aa2715d074b123a93 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:40.375901 19485 tablet_service.cc:1514] Processing DeleteTablet for tablet d63afd081986473aa2715d074b123a93 with delete_type TABLET_DATA_TOMBSTONED (TS 99a0be6ef2144b7fb247ba9d3f051d66 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:37600
I20240501 14:10:40.378515 25766 tablet_replica.cc:330] T d63afd081986473aa2715d074b123a93 P 99a0be6ef2144b7fb247ba9d3f051d66: stopping tablet replica
I20240501 14:10:40.378736 25766 raft_consensus.cc:2229] T d63afd081986473aa2715d074b123a93 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:40.379496 25766 raft_consensus.cc:2258] T d63afd081986473aa2715d074b123a93 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:40.389384 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:40.389822 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:40.405408 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:40.426595 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:40.438915 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:40.458222 19749 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:40.459028 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:40.459336 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:40.461771 25766 ts_tablet_manager.cc:1875] T d63afd081986473aa2715d074b123a93 P 99a0be6ef2144b7fb247ba9d3f051d66: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:40.465860 21247 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:40.467783 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:40.476047 21247 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:40.476503 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:40.487766 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:40.490080 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:40.492692 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:40.506878 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:40.508728 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:40.511454 19617 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:10:40.511926 19617 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:10:40.543352 19617 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:10:40.563980 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:40.569548 21247 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:40.581666 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:40.587746 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:40.590626 25766 ts_tablet_manager.cc:1888] T d63afd081986473aa2715d074b123a93 P 99a0be6ef2144b7fb247ba9d3f051d66: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:10:40.590730 25766 log.cc:1196] T d63afd081986473aa2715d074b123a93 P 99a0be6ef2144b7fb247ba9d3f051d66: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal/wals/d63afd081986473aa2715d074b123a93
I20240501 14:10:40.609081 19219 catalog_manager.cc:4830] TS 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829): tablet d63afd081986473aa2715d074b123a93 (table rebalance_test_table_6 [id=afc36d383e3e4795ad11804e9bebcb51]) successfully deleted
I20240501 14:10:40.674114 19637 raft_consensus.cc:1261] T 12ffa74f7a5b472aac57253693273cad P ec8cde17a00645e7b38d5d29f7425367 [term 2 FOLLOWER]: Refusing update from remote peer e6246b183293412494919f1e7efc9448: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:10:40.676716 25655 consensus_queue.cc:1035] T 12ffa74f7a5b472aac57253693273cad P e6246b183293412494919f1e7efc9448 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:40.696010 19767 consensus_queue.cc:237] T dab5e5722e2e4b0eaea73288d8dce265 P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:40.697396 21137 raft_consensus.cc:1261] T dab5e5722e2e4b0eaea73288d8dce265 P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:40.697396 19506 raft_consensus.cc:1261] T dab5e5722e2e4b0eaea73288d8dce265 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:40.697795 25603 consensus_queue.cc:1035] T dab5e5722e2e4b0eaea73288d8dce265 P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:40.697942 25071 consensus_queue.cc:1035] T dab5e5722e2e4b0eaea73288d8dce265 P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:40.704103 25071 raft_consensus.cc:2932] T dab5e5722e2e4b0eaea73288d8dce265 P da3c4e948ff747678f18456c134d7ed4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:40.705176 21137 raft_consensus.cc:2932] T dab5e5722e2e4b0eaea73288d8dce265 P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:40.708060 19506 raft_consensus.cc:2932] T dab5e5722e2e4b0eaea73288d8dce265 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:40.708988 19228 catalog_manager.cc:5484] T dab5e5722e2e4b0eaea73288d8dce265 P da3c4e948ff747678f18456c134d7ed4 reported cstate change: config changed from index 4 to 5, VOTER ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131) evicted. New cstate: current_term: 1 leader_uuid: "da3c4e948ff747678f18456c134d7ed4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:40.715552 19617 tablet_service.cc:1514] Processing DeleteTablet for tablet dab5e5722e2e4b0eaea73288d8dce265 with delete_type TABLET_DATA_TOMBSTONED (TS ec8cde17a00645e7b38d5d29f7425367 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:47238
I20240501 14:10:40.715829 25671 tablet_replica.cc:330] T dab5e5722e2e4b0eaea73288d8dce265 P ec8cde17a00645e7b38d5d29f7425367: stopping tablet replica
I20240501 14:10:40.715982 25671 raft_consensus.cc:2229] T dab5e5722e2e4b0eaea73288d8dce265 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:40.716159 25671 raft_consensus.cc:2258] T dab5e5722e2e4b0eaea73288d8dce265 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:40.716820 25671 ts_tablet_manager.cc:1875] T dab5e5722e2e4b0eaea73288d8dce265 P ec8cde17a00645e7b38d5d29f7425367: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:40.717454 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:40.720369 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:40.720492 25671 ts_tablet_manager.cc:1888] T dab5e5722e2e4b0eaea73288d8dce265 P ec8cde17a00645e7b38d5d29f7425367: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:10:40.720558 25671 log.cc:1196] T dab5e5722e2e4b0eaea73288d8dce265 P ec8cde17a00645e7b38d5d29f7425367: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal/wals/dab5e5722e2e4b0eaea73288d8dce265
I20240501 14:10:40.727756 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:40.727954 19219 catalog_manager.cc:4830] TS ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201): tablet dab5e5722e2e4b0eaea73288d8dce265 (table rebalance_test_table_1 [id=a9b1c37e923e49a7b30905624977117c]) successfully deleted
I20240501 14:10:40.739621 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:40.747602 21247 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:40.758134 19637 consensus_queue.cc:237] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:10:40.759588 21522 raft_consensus.cc:1261] T fe554c3145de46cf8070efcecffbaedb P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:10:40.760031 21268 raft_consensus.cc:1261] T fe554c3145de46cf8070efcecffbaedb P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:10:40.760263 24775 consensus_queue.cc:1035] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:10:40.760527 24190 consensus_queue.cc:1035] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:10:40.761611 19617 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:10:40.761834 19506 raft_consensus.cc:1261] T fe554c3145de46cf8070efcecffbaedb P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:10:40.762248 24190 consensus_queue.cc:1035] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:10:40.771006 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:40.775333 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:40.775413 19767 raft_consensus.cc:1261] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 [term 2 FOLLOWER]: Refusing update from remote peer e6246b183293412494919f1e7efc9448: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:10:40.775794 25655 consensus_queue.cc:1035] T 12ffa74f7a5b472aac57253693273cad P e6246b183293412494919f1e7efc9448 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:40.777855 24190 raft_consensus.cc:2932] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:40.778158 21522 raft_consensus.cc:2932] T fe554c3145de46cf8070efcecffbaedb P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:40.780678 21268 raft_consensus.cc:2932] T fe554c3145de46cf8070efcecffbaedb P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:40.780967 19230 catalog_manager.cc:5484] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 reported cstate change: config changed from index 5 to 6, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:40.785770 19748 tablet_service.cc:1514] Processing DeleteTablet for tablet fe554c3145de46cf8070efcecffbaedb with delete_type TABLET_DATA_TOMBSTONED (TS da3c4e948ff747678f18456c134d7ed4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36530
I20240501 14:10:40.786438 25672 tablet_replica.cc:330] T fe554c3145de46cf8070efcecffbaedb P da3c4e948ff747678f18456c134d7ed4: stopping tablet replica
I20240501 14:10:40.786579 25672 raft_consensus.cc:2229] T fe554c3145de46cf8070efcecffbaedb P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:40.786737 25672 raft_consensus.cc:2258] T fe554c3145de46cf8070efcecffbaedb P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:40.787292 25672 ts_tablet_manager.cc:1875] T fe554c3145de46cf8070efcecffbaedb P da3c4e948ff747678f18456c134d7ed4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:40.789510 25672 ts_tablet_manager.cc:1888] T fe554c3145de46cf8070efcecffbaedb P da3c4e948ff747678f18456c134d7ed4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:10:40.789580 25672 log.cc:1196] T fe554c3145de46cf8070efcecffbaedb P da3c4e948ff747678f18456c134d7ed4: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal/wals/fe554c3145de46cf8070efcecffbaedb
I20240501 14:10:40.791980 19220 catalog_manager.cc:4830] TS da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357): tablet fe554c3145de46cf8070efcecffbaedb (table rebalance_test_table_9 [id=1fbbd460d13647c1a85b517926267158]) successfully deleted
I20240501 14:10:40.792600 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:40.801827 21247 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:40.805806 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:40.806777 19375 raft_consensus.cc:1261] T 12ffa74f7a5b472aac57253693273cad P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 2 FOLLOWER]: Refusing update from remote peer e6246b183293412494919f1e7efc9448: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:10:40.811151 25655 consensus_queue.cc:1035] T 12ffa74f7a5b472aac57253693273cad P e6246b183293412494919f1e7efc9448 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:40.819852 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:40.824249 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:40.834810 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:40.844144 19617 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:10:40.848611 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:40.884093 19506 raft_consensus.cc:2932] T fe554c3145de46cf8070efcecffbaedb P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:40.892400 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:40.895366 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:40.906186 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:40.908998 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:40.923645 21247 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:40.930457 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:40.934597 19617 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:10:40.955940 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:41.002532 19637 consensus_queue.cc:237] T dadab886b03540579d138cdb5b25e286 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:10:41.004066 19375 raft_consensus.cc:1261] T dadab886b03540579d138cdb5b25e286 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:41.004552 21522 raft_consensus.cc:1261] T dadab886b03540579d138cdb5b25e286 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:41.005121 25959 consensus_queue.cc:1035] T dadab886b03540579d138cdb5b25e286 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:41.005465 25959 consensus_queue.cc:1035] T dadab886b03540579d138cdb5b25e286 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:41.017824 24190 raft_consensus.cc:2932] T dadab886b03540579d138cdb5b25e286 P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:41.020638 19230 catalog_manager.cc:5484] T dadab886b03540579d138cdb5b25e286 P ec8cde17a00645e7b38d5d29f7425367 reported cstate change: config changed from index 4 to 5, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:41.021147 19375 raft_consensus.cc:2932] T dadab886b03540579d138cdb5b25e286 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:41.025228 19485 tablet_service.cc:1514] Processing DeleteTablet for tablet dadab886b03540579d138cdb5b25e286 with delete_type TABLET_DATA_TOMBSTONED (TS 99a0be6ef2144b7fb247ba9d3f051d66 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:37600
I20240501 14:10:41.025507 25766 tablet_replica.cc:330] T dadab886b03540579d138cdb5b25e286 P 99a0be6ef2144b7fb247ba9d3f051d66: stopping tablet replica
I20240501 14:10:41.025678 25766 raft_consensus.cc:2229] T dadab886b03540579d138cdb5b25e286 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:41.025841 25766 raft_consensus.cc:2258] T dadab886b03540579d138cdb5b25e286 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:41.026495 25766 ts_tablet_manager.cc:1875] T dadab886b03540579d138cdb5b25e286 P 99a0be6ef2144b7fb247ba9d3f051d66: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:41.028849 25766 ts_tablet_manager.cc:1888] T dadab886b03540579d138cdb5b25e286 P 99a0be6ef2144b7fb247ba9d3f051d66: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:10:41.028913 25766 log.cc:1196] T dadab886b03540579d138cdb5b25e286 P 99a0be6ef2144b7fb247ba9d3f051d66: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal/wals/dadab886b03540579d138cdb5b25e286
I20240501 14:10:41.029580 19219 catalog_manager.cc:4830] TS 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829): tablet dadab886b03540579d138cdb5b25e286 (table rebalance_test_table_1 [id=a9b1c37e923e49a7b30905624977117c]) successfully deleted
I20240501 14:10:41.043684 21522 raft_consensus.cc:2932] T dadab886b03540579d138cdb5b25e286 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:41.044178 19637 consensus_queue.cc:237] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:41.045501 21268 raft_consensus.cc:1261] T 85593473500b4d5eaf9f8b0bf2296777 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:10:41.046066 25959 consensus_queue.cc:1035] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:10:41.049300 25959 raft_consensus.cc:2932] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:41.051191 21268 raft_consensus.cc:2932] T 85593473500b4d5eaf9f8b0bf2296777 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:41.054184 19229 catalog_manager.cc:5484] T 85593473500b4d5eaf9f8b0bf2296777 P e6246b183293412494919f1e7efc9448 reported cstate change: config changed from index 5 to 6, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) evicted. New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:41.059010 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:41.059965 19637 consensus_queue.cc:237] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:41.061208 21268 raft_consensus.cc:1261] T 85593473500b4d5eaf9f8b0bf2296777 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 14:10:41.061794 19571 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135:38751): Couldn't send request to peer 43ce4a4c954f4a5f80954b38933c189d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 85593473500b4d5eaf9f8b0bf2296777. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:41.062179 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:41.062268 24190 consensus_queue.cc:1035] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:10:41.063546 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:41.064613 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:41.065917 21247 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:41.067426 19356 tablet_service.cc:1514] Processing DeleteTablet for tablet 85593473500b4d5eaf9f8b0bf2296777 with delete_type TABLET_DATA_TOMBSTONED (TS fb1e6bc2d07b4ce3b622a66f14c93bdc not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46838
I20240501 14:10:41.076469 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:41.084394 24190 raft_consensus.cc:2932] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:41.087121 19219 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 85593473500b4d5eaf9f8b0bf2296777 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:10:41.091538 21268 raft_consensus.cc:2932] T 85593473500b4d5eaf9f8b0bf2296777 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:41.093302 19617 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:10:41.094615 19229 catalog_manager.cc:5484] T 85593473500b4d5eaf9f8b0bf2296777 P e6246b183293412494919f1e7efc9448 reported cstate change: config changed from index 6 to 7, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:41.108290 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:41.116189 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:41.123310 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:41.125721 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:41.126724 26122 tablet_replica.cc:330] T 85593473500b4d5eaf9f8b0bf2296777 P fb1e6bc2d07b4ce3b622a66f14c93bdc: stopping tablet replica
I20240501 14:10:41.126914 26122 raft_consensus.cc:2229] T 85593473500b4d5eaf9f8b0bf2296777 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:41.127107 26122 raft_consensus.cc:2258] T 85593473500b4d5eaf9f8b0bf2296777 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:41.129104 26122 ts_tablet_manager.cc:1875] T 85593473500b4d5eaf9f8b0bf2296777 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:41.133782 26122 ts_tablet_manager.cc:1888] T 85593473500b4d5eaf9f8b0bf2296777 P fb1e6bc2d07b4ce3b622a66f14c93bdc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:10:41.134788 26122 log.cc:1196] T 85593473500b4d5eaf9f8b0bf2296777 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal/wals/85593473500b4d5eaf9f8b0bf2296777
I20240501 14:10:41.139816 19218 catalog_manager.cc:4830] TS fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847): tablet 85593473500b4d5eaf9f8b0bf2296777 (table rebalance_test_table_0 [id=b52269ef7c494f74a5e4f0d5cb3b8a47]) successfully deleted
I20240501 14:10:41.142666 21247 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:41.148350 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:41.157020 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:41.157287 19506 consensus_queue.cc:237] T a9edf1764b55420ea5d519e07ae1d5e0 P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:10:41.158584 19637 raft_consensus.cc:1261] T a9edf1764b55420ea5d519e07ae1d5e0 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Refusing update from remote peer 99a0be6ef2144b7fb247ba9d3f051d66: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:41.158967 26138 consensus_queue.cc:1035] T a9edf1764b55420ea5d519e07ae1d5e0 P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:41.159708 21522 raft_consensus.cc:1261] T a9edf1764b55420ea5d519e07ae1d5e0 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Refusing update from remote peer 99a0be6ef2144b7fb247ba9d3f051d66: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:41.160169 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:41.160991 26138 consensus_queue.cc:1035] T a9edf1764b55420ea5d519e07ae1d5e0 P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:41.171671 19617 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:10:41.197355 26138 raft_consensus.cc:2932] T a9edf1764b55420ea5d519e07ae1d5e0 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:41.200143 19637 raft_consensus.cc:2932] T a9edf1764b55420ea5d519e07ae1d5e0 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:41.200433 19230 catalog_manager.cc:5484] T a9edf1764b55420ea5d519e07ae1d5e0 P 99a0be6ef2144b7fb247ba9d3f051d66 reported cstate change: config changed from index 4 to 5, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New cstate: current_term: 1 leader_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:41.204321 19748 tablet_service.cc:1514] Processing DeleteTablet for tablet a9edf1764b55420ea5d519e07ae1d5e0 with delete_type TABLET_DATA_TOMBSTONED (TS da3c4e948ff747678f18456c134d7ed4 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:36530
I20240501 14:10:41.204632 25672 tablet_replica.cc:330] T a9edf1764b55420ea5d519e07ae1d5e0 P da3c4e948ff747678f18456c134d7ed4: stopping tablet replica
I20240501 14:10:41.204818 25672 raft_consensus.cc:2229] T a9edf1764b55420ea5d519e07ae1d5e0 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:41.204962 25672 raft_consensus.cc:2258] T a9edf1764b55420ea5d519e07ae1d5e0 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:41.205523 25672 ts_tablet_manager.cc:1875] T a9edf1764b55420ea5d519e07ae1d5e0 P da3c4e948ff747678f18456c134d7ed4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:41.209440 25672 ts_tablet_manager.cc:1888] T a9edf1764b55420ea5d519e07ae1d5e0 P da3c4e948ff747678f18456c134d7ed4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:10:41.209606 25672 log.cc:1196] T a9edf1764b55420ea5d519e07ae1d5e0 P da3c4e948ff747678f18456c134d7ed4: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal/wals/a9edf1764b55420ea5d519e07ae1d5e0
I20240501 14:10:41.210388 19220 catalog_manager.cc:4830] TS da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357): tablet a9edf1764b55420ea5d519e07ae1d5e0 (table rebalance_test_table_6 [id=afc36d383e3e4795ad11804e9bebcb51]) successfully deleted
I20240501 14:10:41.235884 21522 raft_consensus.cc:2932] T a9edf1764b55420ea5d519e07ae1d5e0 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:41.276746 19637 consensus_queue.cc:237] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:41.278259 21268 raft_consensus.cc:1261] T 661aaf83e3474966a257fcc039ef4784 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:41.278323 19769 raft_consensus.cc:1261] T 661aaf83e3474966a257fcc039ef4784 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:41.279004 24775 consensus_queue.cc:1035] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:41.279230 25959 consensus_queue.cc:1035] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:41.280889 24775 raft_consensus.cc:2932] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:41.283049 19375 consensus_queue.cc:237] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:41.283139 21268 raft_consensus.cc:2932] T 661aaf83e3474966a257fcc039ef4784 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:41.286245 19506 raft_consensus.cc:1261] T 814ba2247a314a4797aa0f9f624aaeb6 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:41.286358 21268 raft_consensus.cc:1261] T 814ba2247a314a4797aa0f9f624aaeb6 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:41.286502 19229 catalog_manager.cc:5484] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 reported cstate change: config changed from index 4 to 5, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:41.287412 25636 consensus_queue.cc:1035] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:41.287909 25636 consensus_queue.cc:1035] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:41.291270 19485 tablet_service.cc:1514] Processing DeleteTablet for tablet 661aaf83e3474966a257fcc039ef4784 with delete_type TABLET_DATA_TOMBSTONED (TS 99a0be6ef2144b7fb247ba9d3f051d66 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:37600
I20240501 14:10:41.291501 25766 tablet_replica.cc:330] T 661aaf83e3474966a257fcc039ef4784 P 99a0be6ef2144b7fb247ba9d3f051d66: stopping tablet replica
I20240501 14:10:41.291633 25766 raft_consensus.cc:2229] T 661aaf83e3474966a257fcc039ef4784 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:41.291769 25766 raft_consensus.cc:2258] T 661aaf83e3474966a257fcc039ef4784 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:41.292315 25766 ts_tablet_manager.cc:1875] T 661aaf83e3474966a257fcc039ef4784 P 99a0be6ef2144b7fb247ba9d3f051d66: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:41.290428 26251 raft_consensus.cc:2932] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:41.294513 25766 ts_tablet_manager.cc:1888] T 661aaf83e3474966a257fcc039ef4784 P 99a0be6ef2144b7fb247ba9d3f051d66: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:10:41.294584 25766 log.cc:1196] T 661aaf83e3474966a257fcc039ef4784 P 99a0be6ef2144b7fb247ba9d3f051d66: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal/wals/661aaf83e3474966a257fcc039ef4784
I20240501 14:10:41.295526 19219 catalog_manager.cc:4830] TS 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829): tablet 661aaf83e3474966a257fcc039ef4784 (table rebalance_test_table_1 [id=a9b1c37e923e49a7b30905624977117c]) successfully deleted
I20240501 14:10:41.295428 21268 raft_consensus.cc:2932] T 814ba2247a314a4797aa0f9f624aaeb6 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:41.297298 19506 raft_consensus.cc:2932] T 814ba2247a314a4797aa0f9f624aaeb6 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:41.298087 19769 raft_consensus.cc:2932] T 661aaf83e3474966a257fcc039ef4784 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:41.299739 19225 catalog_manager.cc:5484] T 814ba2247a314a4797aa0f9f624aaeb6 P e6246b183293412494919f1e7efc9448 reported cstate change: config changed from index 4 to 5, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New cstate: current_term: 1 leader_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:41.302357 19748 tablet_service.cc:1514] Processing DeleteTablet for tablet 814ba2247a314a4797aa0f9f624aaeb6 with delete_type TABLET_DATA_TOMBSTONED (TS da3c4e948ff747678f18456c134d7ed4 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:36530
I20240501 14:10:41.302561 25672 tablet_replica.cc:330] T 814ba2247a314a4797aa0f9f624aaeb6 P da3c4e948ff747678f18456c134d7ed4: stopping tablet replica
I20240501 14:10:41.302686 25672 raft_consensus.cc:2229] T 814ba2247a314a4797aa0f9f624aaeb6 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:41.302819 25672 raft_consensus.cc:2258] T 814ba2247a314a4797aa0f9f624aaeb6 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:41.303344 25672 ts_tablet_manager.cc:1875] T 814ba2247a314a4797aa0f9f624aaeb6 P da3c4e948ff747678f18456c134d7ed4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:41.305583 25672 ts_tablet_manager.cc:1888] T 814ba2247a314a4797aa0f9f624aaeb6 P da3c4e948ff747678f18456c134d7ed4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:10:41.305652 25672 log.cc:1196] T 814ba2247a314a4797aa0f9f624aaeb6 P da3c4e948ff747678f18456c134d7ed4: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal/wals/814ba2247a314a4797aa0f9f624aaeb6
I20240501 14:10:41.306162 19220 catalog_manager.cc:4830] TS da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357): tablet 814ba2247a314a4797aa0f9f624aaeb6 (table rebalance_test_table_4 [id=151dfad0ef40470ab9c5203c42400b60]) successfully deleted
I20240501 14:10:41.331524 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:41.335847 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:41.337344 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:41.338282 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:41.349320 21247 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:41.359215 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:41.376103 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:41.395200 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:41.401295 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:41.403963 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:41.419991 19617 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:10:41.440614 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:41.442103 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:41.443938 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:41.450467 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:41.481647 21247 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:41.484520 21247 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:41.488590 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:41.490495 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:41.496385 19617 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:10:41.509198 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:41.509747 19617 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:10:41.526916 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:41.540956 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:41.573719 26368 ts_tablet_manager.cc:902] T 85593473500b4d5eaf9f8b0bf2296777 P 43ce4a4c954f4a5f80954b38933c189d: Initiating tablet copy from peer ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201)
I20240501 14:10:41.574258 26368 tablet_copy_client.cc:308] T 85593473500b4d5eaf9f8b0bf2296777 P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.131:45201
I20240501 14:10:41.574802 19657 tablet_copy_service.cc:140] P ec8cde17a00645e7b38d5d29f7425367: Received BeginTabletCopySession request for tablet 85593473500b4d5eaf9f8b0bf2296777 from peer 43ce4a4c954f4a5f80954b38933c189d ({username='slave'} at 127.7.39.135:35759)
I20240501 14:10:41.574903 19657 tablet_copy_service.cc:161] P ec8cde17a00645e7b38d5d29f7425367: Beginning new tablet copy session on tablet 85593473500b4d5eaf9f8b0bf2296777 from peer 43ce4a4c954f4a5f80954b38933c189d at {username='slave'} at 127.7.39.135:35759: session id = 43ce4a4c954f4a5f80954b38933c189d-85593473500b4d5eaf9f8b0bf2296777
I20240501 14:10:41.575963 19657 tablet_copy_source_session.cc:205] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:41.584105 26368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85593473500b4d5eaf9f8b0bf2296777. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:41.586814 26368 tablet_copy_client.cc:783] T 85593473500b4d5eaf9f8b0bf2296777 P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:41.587019 26368 tablet_copy_client.cc:647] T 85593473500b4d5eaf9f8b0bf2296777 P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:41.589547 26368 tablet_copy_client.cc:523] T 85593473500b4d5eaf9f8b0bf2296777 P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:41.595309 26368 tablet_bootstrap.cc:492] T 85593473500b4d5eaf9f8b0bf2296777 P 43ce4a4c954f4a5f80954b38933c189d: Bootstrap starting.
I20240501 14:10:41.627831 26368 tablet_bootstrap.cc:492] T 85593473500b4d5eaf9f8b0bf2296777 P 43ce4a4c954f4a5f80954b38933c189d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:41.628135 26368 tablet_bootstrap.cc:492] T 85593473500b4d5eaf9f8b0bf2296777 P 43ce4a4c954f4a5f80954b38933c189d: Bootstrap complete.
I20240501 14:10:41.628387 26368 ts_tablet_manager.cc:1367] T 85593473500b4d5eaf9f8b0bf2296777 P 43ce4a4c954f4a5f80954b38933c189d: Time spent bootstrapping tablet: real 0.033s user 0.028s sys 0.000s
I20240501 14:10:41.628850 26368 raft_consensus.cc:348] T 85593473500b4d5eaf9f8b0bf2296777 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:41.628954 26368 raft_consensus.cc:729] T 85593473500b4d5eaf9f8b0bf2296777 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43ce4a4c954f4a5f80954b38933c189d, State: Initialized, Role: FOLLOWER
I20240501 14:10:41.629455 26368 consensus_queue.cc:260] T 85593473500b4d5eaf9f8b0bf2296777 P 43ce4a4c954f4a5f80954b38933c189d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:41.629884 26368 ts_tablet_manager.cc:1398] T 85593473500b4d5eaf9f8b0bf2296777 P 43ce4a4c954f4a5f80954b38933c189d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:10:41.630399 19657 tablet_copy_service.cc:342] P ec8cde17a00645e7b38d5d29f7425367: Request end of tablet copy session 43ce4a4c954f4a5f80954b38933c189d-85593473500b4d5eaf9f8b0bf2296777 received from {username='slave'} at 127.7.39.135:35759
I20240501 14:10:41.630486 19657 tablet_copy_service.cc:433] P ec8cde17a00645e7b38d5d29f7425367: ending tablet copy session 43ce4a4c954f4a5f80954b38933c189d-85593473500b4d5eaf9f8b0bf2296777 on tablet 85593473500b4d5eaf9f8b0bf2296777 with peer 43ce4a4c954f4a5f80954b38933c189d
I20240501 14:10:41.687875 19637 consensus_queue.cc:237] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:41.688433 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:41.689508 21268 raft_consensus.cc:1261] T 2dcca6e0e6374520bd4522c80c110e97 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:10:41.689986 25959 consensus_queue.cc:1035] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:10:41.691267 21399 raft_consensus.cc:1261] T 2dcca6e0e6374520bd4522c80c110e97 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:10:41.691721 25959 consensus_queue.cc:1035] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:10:41.696403 21268 raft_consensus.cc:2932] T 2dcca6e0e6374520bd4522c80c110e97 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:41.698390 24775 raft_consensus.cc:2932] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:41.699766 19227 catalog_manager.cc:5484] T 2dcca6e0e6374520bd4522c80c110e97 P e6246b183293412494919f1e7efc9448 reported cstate change: config changed from index 6 to 7, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:41.702394 19485 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dcca6e0e6374520bd4522c80c110e97 with delete_type TABLET_DATA_TOMBSTONED (TS 99a0be6ef2144b7fb247ba9d3f051d66 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:37600
I20240501 14:10:41.702626 25766 tablet_replica.cc:330] T 2dcca6e0e6374520bd4522c80c110e97 P 99a0be6ef2144b7fb247ba9d3f051d66: stopping tablet replica
I20240501 14:10:41.702790 25766 raft_consensus.cc:2229] T 2dcca6e0e6374520bd4522c80c110e97 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:41.702989 25766 raft_consensus.cc:2258] T 2dcca6e0e6374520bd4522c80c110e97 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:41.703598 25766 ts_tablet_manager.cc:1875] T 2dcca6e0e6374520bd4522c80c110e97 P 99a0be6ef2144b7fb247ba9d3f051d66: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:41.704633 21399 raft_consensus.cc:2932] T 2dcca6e0e6374520bd4522c80c110e97 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:41.705822 25766 ts_tablet_manager.cc:1888] T 2dcca6e0e6374520bd4522c80c110e97 P 99a0be6ef2144b7fb247ba9d3f051d66: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:10:41.705893 25766 log.cc:1196] T 2dcca6e0e6374520bd4522c80c110e97 P 99a0be6ef2144b7fb247ba9d3f051d66: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal/wals/2dcca6e0e6374520bd4522c80c110e97
I20240501 14:10:41.706763 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:41.707815 19219 catalog_manager.cc:4830] TS 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829): tablet 2dcca6e0e6374520bd4522c80c110e97 (table rebalance_test_table_8 [id=3866c35dd0e64d87b6049e5e14cb0d31]) successfully deleted
I20240501 14:10:41.710765 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:41.712378 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:41.747357 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:41.748409 21247 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:41.756705 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:41.759454 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:41.766382 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:41.766484 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:41.771499 19617 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:10:41.779783 21247 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:41.782792 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:41.785065 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:41.789232 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:41.790263 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:41.800961 19376 consensus_queue.cc:237] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:41.802593 21268 raft_consensus.cc:1261] T 814ba2247a314a4797aa0f9f624aaeb6 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:10:41.802991 26252 consensus_queue.cc:1035] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:10:41.812834 26252 raft_consensus.cc:2932] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:41.817178 21268 raft_consensus.cc:2932] T 814ba2247a314a4797aa0f9f624aaeb6 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:41.817503 19229 catalog_manager.cc:5484] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc reported cstate change: config changed from index 5 to 6, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New cstate: current_term: 1 leader_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:41.819928 21247 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:41.822154 19485 tablet_service.cc:1514] Processing DeleteTablet for tablet 814ba2247a314a4797aa0f9f624aaeb6 with delete_type TABLET_DATA_TOMBSTONED (TS 99a0be6ef2144b7fb247ba9d3f051d66 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37600
I20240501 14:10:41.822371 25766 tablet_replica.cc:330] T 814ba2247a314a4797aa0f9f624aaeb6 P 99a0be6ef2144b7fb247ba9d3f051d66: stopping tablet replica
I20240501 14:10:41.822513 25766 raft_consensus.cc:2229] T 814ba2247a314a4797aa0f9f624aaeb6 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:41.822726 25766 raft_consensus.cc:2258] T 814ba2247a314a4797aa0f9f624aaeb6 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:41.823694 25766 ts_tablet_manager.cc:1875] T 814ba2247a314a4797aa0f9f624aaeb6 P 99a0be6ef2144b7fb247ba9d3f051d66: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:41.824577 19376 consensus_queue.cc:237] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
W20240501 14:10:41.826037 19309 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135:38751): Couldn't send request to peer 43ce4a4c954f4a5f80954b38933c189d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 814ba2247a314a4797aa0f9f624aaeb6. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:41.826115 25766 ts_tablet_manager.cc:1888] T 814ba2247a314a4797aa0f9f624aaeb6 P 99a0be6ef2144b7fb247ba9d3f051d66: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:10:41.826135 21268 raft_consensus.cc:1261] T 814ba2247a314a4797aa0f9f624aaeb6 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:10:41.826234 25766 log.cc:1196] T 814ba2247a314a4797aa0f9f624aaeb6 P 99a0be6ef2144b7fb247ba9d3f051d66: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal/wals/814ba2247a314a4797aa0f9f624aaeb6
I20240501 14:10:41.826489 25636 consensus_queue.cc:1035] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:10:41.826872 19219 catalog_manager.cc:4830] TS 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829): tablet 814ba2247a314a4797aa0f9f624aaeb6 (table rebalance_test_table_4 [id=151dfad0ef40470ab9c5203c42400b60]) successfully deleted
I20240501 14:10:41.827607 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:41.845628 19617 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:10:41.846215 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:41.850289 19617 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:10:41.850524 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:41.859618 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:41.867770 25636 raft_consensus.cc:2932] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:41.869987 19218 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 814ba2247a314a4797aa0f9f624aaeb6 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:10:41.870529 21268 raft_consensus.cc:2932] T 814ba2247a314a4797aa0f9f624aaeb6 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:41.870699 19225 catalog_manager.cc:5484] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc reported cstate change: config changed from index 6 to 7, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New cstate: current_term: 1 leader_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:41.886159 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:41.962390 19637 consensus_queue.cc:237] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:10:41.964151 21522 raft_consensus.cc:1261] T fe554c3145de46cf8070efcecffbaedb P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:10:41.964728 24775 consensus_queue.cc:1035] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:10:41.964955 21268 raft_consensus.cc:1261] T fe554c3145de46cf8070efcecffbaedb P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:10:41.965476 24775 consensus_queue.cc:1035] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:10:41.968525 24775 raft_consensus.cc:2932] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:41.970659 21268 raft_consensus.cc:2932] T fe554c3145de46cf8070efcecffbaedb P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:41.971647 19225 catalog_manager.cc:5484] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 reported cstate change: config changed from index 6 to 7, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:41.975889 19485 tablet_service.cc:1514] Processing DeleteTablet for tablet fe554c3145de46cf8070efcecffbaedb with delete_type TABLET_DATA_TOMBSTONED (TS 99a0be6ef2144b7fb247ba9d3f051d66 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:37600
I20240501 14:10:41.976135 25766 tablet_replica.cc:330] T fe554c3145de46cf8070efcecffbaedb P 99a0be6ef2144b7fb247ba9d3f051d66: stopping tablet replica
I20240501 14:10:41.976302 25766 raft_consensus.cc:2229] T fe554c3145de46cf8070efcecffbaedb P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:41.976487 25766 raft_consensus.cc:2258] T fe554c3145de46cf8070efcecffbaedb P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:41.977095 25766 ts_tablet_manager.cc:1875] T fe554c3145de46cf8070efcecffbaedb P 99a0be6ef2144b7fb247ba9d3f051d66: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:41.979969 25766 ts_tablet_manager.cc:1888] T fe554c3145de46cf8070efcecffbaedb P 99a0be6ef2144b7fb247ba9d3f051d66: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:10:41.980052 25766 log.cc:1196] T fe554c3145de46cf8070efcecffbaedb P 99a0be6ef2144b7fb247ba9d3f051d66: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal/wals/fe554c3145de46cf8070efcecffbaedb
I20240501 14:10:41.981132 19219 catalog_manager.cc:4830] TS 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829): tablet fe554c3145de46cf8070efcecffbaedb (table rebalance_test_table_9 [id=1fbbd460d13647c1a85b517926267158]) successfully deleted
I20240501 14:10:41.984218 19769 consensus_queue.cc:237] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:10:41.985931 21137 raft_consensus.cc:1261] T ec2a46f8cfb3406082970e5fdb59b101 P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:10:41.986135 19376 raft_consensus.cc:1261] T ec2a46f8cfb3406082970e5fdb59b101 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:10:41.986387 21530 raft_consensus.cc:1261] T ec2a46f8cfb3406082970e5fdb59b101 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:10:41.986977 25934 consensus_queue.cc:1035] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:10:41.987347 25934 consensus_queue.cc:1035] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:10:41.987629 25934 consensus_queue.cc:1035] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:10:41.992077 19637 consensus_queue.cc:237] T 10aa2feeb0324aa4a6221d2f2f3fc941 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:41.992146 25934 raft_consensus.cc:2932] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:41.994197 19376 raft_consensus.cc:1261] T 10aa2feeb0324aa4a6221d2f2f3fc941 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:10:41.994654 21530 raft_consensus.cc:1261] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:10:41.994724 21399 raft_consensus.cc:1261] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:10:41.994760 24775 consensus_queue.cc:1035] T 10aa2feeb0324aa4a6221d2f2f3fc941 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:10:41.995254 26475 consensus_queue.cc:1035] T 10aa2feeb0324aa4a6221d2f2f3fc941 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:10:41.995733 21530 raft_consensus.cc:2932] T ec2a46f8cfb3406082970e5fdb59b101 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:41.996927 19229 catalog_manager.cc:5484] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4 reported cstate change: config changed from index 5 to 6, VOTER ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131) evicted. New cstate: current_term: 1 leader_uuid: "da3c4e948ff747678f18456c134d7ed4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:41.997711 19376 raft_consensus.cc:2932] T ec2a46f8cfb3406082970e5fdb59b101 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:41.999723 19617 tablet_service.cc:1514] Processing DeleteTablet for tablet ec2a46f8cfb3406082970e5fdb59b101 with delete_type TABLET_DATA_TOMBSTONED (TS ec8cde17a00645e7b38d5d29f7425367 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47238
I20240501 14:10:41.999967 24775 consensus_queue.cc:1035] T 10aa2feeb0324aa4a6221d2f2f3fc941 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:10:42.002218 21530 raft_consensus.cc:2932] T fe554c3145de46cf8070efcecffbaedb P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:42.002833 26624 tablet_replica.cc:330] T ec2a46f8cfb3406082970e5fdb59b101 P ec8cde17a00645e7b38d5d29f7425367: stopping tablet replica
I20240501 14:10:42.002977 26624 raft_consensus.cc:2229] T ec2a46f8cfb3406082970e5fdb59b101 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:42.003155 26624 raft_consensus.cc:2258] T ec2a46f8cfb3406082970e5fdb59b101 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:42.004212 26624 ts_tablet_manager.cc:1875] T ec2a46f8cfb3406082970e5fdb59b101 P ec8cde17a00645e7b38d5d29f7425367: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:42.007680 26624 ts_tablet_manager.cc:1888] T ec2a46f8cfb3406082970e5fdb59b101 P ec8cde17a00645e7b38d5d29f7425367: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:10:42.007854 26624 log.cc:1196] T ec2a46f8cfb3406082970e5fdb59b101 P ec8cde17a00645e7b38d5d29f7425367: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal/wals/ec2a46f8cfb3406082970e5fdb59b101
I20240501 14:10:42.008793 19219 catalog_manager.cc:4830] TS ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201): tablet ec2a46f8cfb3406082970e5fdb59b101 (table rebalance_test_table_3 [id=35d6f8fac5394197b987aa44dab206c1]) successfully deleted
I20240501 14:10:42.009603 21137 raft_consensus.cc:2932] T ec2a46f8cfb3406082970e5fdb59b101 P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:42.013741 24775 raft_consensus.cc:2932] T 10aa2feeb0324aa4a6221d2f2f3fc941 P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:42.016566 21522 raft_consensus.cc:2932] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:42.016814 19375 raft_consensus.cc:2932] T 10aa2feeb0324aa4a6221d2f2f3fc941 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:42.019344 19225 catalog_manager.cc:5484] T 10aa2feeb0324aa4a6221d2f2f3fc941 P ec8cde17a00645e7b38d5d29f7425367 reported cstate change: config changed from index 5 to 6, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:42.021688 21399 raft_consensus.cc:2932] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:42.025305 19485 tablet_service.cc:1514] Processing DeleteTablet for tablet 10aa2feeb0324aa4a6221d2f2f3fc941 with delete_type TABLET_DATA_TOMBSTONED (TS 99a0be6ef2144b7fb247ba9d3f051d66 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37600
I20240501 14:10:42.025646 25766 tablet_replica.cc:330] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 99a0be6ef2144b7fb247ba9d3f051d66: stopping tablet replica
I20240501 14:10:42.025794 25766 raft_consensus.cc:2229] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:42.025971 25766 raft_consensus.cc:2258] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:42.026499 25766 ts_tablet_manager.cc:1875] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 99a0be6ef2144b7fb247ba9d3f051d66: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:42.028756 25766 ts_tablet_manager.cc:1888] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 99a0be6ef2144b7fb247ba9d3f051d66: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:10:42.033358 25766 log.cc:1196] T 10aa2feeb0324aa4a6221d2f2f3fc941 P 99a0be6ef2144b7fb247ba9d3f051d66: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal/wals/10aa2feeb0324aa4a6221d2f2f3fc941
I20240501 14:10:42.033972 19219 catalog_manager.cc:4830] TS 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829): tablet 10aa2feeb0324aa4a6221d2f2f3fc941 (table rebalance_test_table_9 [id=1fbbd460d13647c1a85b517926267158]) successfully deleted
I20240501 14:10:42.090974 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:42.096184 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:42.104777 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:42.106151 19617 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:10:42.109630 21247 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:42.112196 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:42.133301 21398 raft_consensus.cc:1203] T 85593473500b4d5eaf9f8b0bf2296777 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:10:42.143625 21494 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:42.144117 21492 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:42.152603 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:42.162855 21247 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:42.171905 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:42.177096 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:42.179567 19617 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:10:42.183152 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:42.195214 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:42.205190 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:42.287598 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:42.291160 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:42.296275 21492 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:42.300068 21247 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:42.300704 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:42.313879 19617 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:10:42.317827 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:42.348629 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:42.373615 19767 consensus_queue.cc:237] T bb291a0c743c4e638f79da36cc77faa3 P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:42.375192 21522 raft_consensus.cc:1261] T bb291a0c743c4e638f79da36cc77faa3 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:10:42.375754 21137 raft_consensus.cc:1261] T bb291a0c743c4e638f79da36cc77faa3 P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:10:42.375845 26603 consensus_queue.cc:1035] T bb291a0c743c4e638f79da36cc77faa3 P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:10:42.376184 25934 consensus_queue.cc:1035] T bb291a0c743c4e638f79da36cc77faa3 P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:10:42.382326 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:42.391068 21492 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:42.399717 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:42.401934 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:42.409312 19617 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:10:42.410460 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:42.424360 26602 raft_consensus.cc:2932] T bb291a0c743c4e638f79da36cc77faa3 P da3c4e948ff747678f18456c134d7ed4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:42.426867 26882 ts_tablet_manager.cc:902] T 814ba2247a314a4797aa0f9f624aaeb6 P 43ce4a4c954f4a5f80954b38933c189d: Initiating tablet copy from peer fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847)
I20240501 14:10:42.427546 26882 tablet_copy_client.cc:308] T 814ba2247a314a4797aa0f9f624aaeb6 P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.129:35847
I20240501 14:10:42.427610 21247 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:42.428059 19396 tablet_copy_service.cc:140] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Received BeginTabletCopySession request for tablet 814ba2247a314a4797aa0f9f624aaeb6 from peer 43ce4a4c954f4a5f80954b38933c189d ({username='slave'} at 127.7.39.135:49533)
I20240501 14:10:42.428187 19396 tablet_copy_service.cc:161] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Beginning new tablet copy session on tablet 814ba2247a314a4797aa0f9f624aaeb6 from peer 43ce4a4c954f4a5f80954b38933c189d at {username='slave'} at 127.7.39.135:49533: session id = 43ce4a4c954f4a5f80954b38933c189d-814ba2247a314a4797aa0f9f624aaeb6
I20240501 14:10:42.429273 19396 tablet_copy_source_session.cc:205] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:42.429932 19232 catalog_manager.cc:5484] T bb291a0c743c4e638f79da36cc77faa3 P da3c4e948ff747678f18456c134d7ed4 reported cstate change: config changed from index 6 to 7, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) evicted. New cstate: current_term: 1 leader_uuid: "da3c4e948ff747678f18456c134d7ed4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:42.430279 21522 raft_consensus.cc:2932] T bb291a0c743c4e638f79da36cc77faa3 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:42.432833 19356 tablet_service.cc:1514] Processing DeleteTablet for tablet bb291a0c743c4e638f79da36cc77faa3 with delete_type TABLET_DATA_TOMBSTONED (TS fb1e6bc2d07b4ce3b622a66f14c93bdc not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:46838
I20240501 14:10:42.436406 26882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 814ba2247a314a4797aa0f9f624aaeb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:42.437500 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:42.439389 21136 raft_consensus.cc:2932] T bb291a0c743c4e638f79da36cc77faa3 P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:42.443682 26882 tablet_copy_client.cc:783] T 814ba2247a314a4797aa0f9f624aaeb6 P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:42.443887 26882 tablet_copy_client.cc:647] T 814ba2247a314a4797aa0f9f624aaeb6 P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:42.454376 26882 tablet_copy_client.cc:523] T 814ba2247a314a4797aa0f9f624aaeb6 P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:42.456089 26882 tablet_bootstrap.cc:492] T 814ba2247a314a4797aa0f9f624aaeb6 P 43ce4a4c954f4a5f80954b38933c189d: Bootstrap starting.
I20240501 14:10:42.462471 21492 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:42.465931 26887 tablet_replica.cc:330] T bb291a0c743c4e638f79da36cc77faa3 P fb1e6bc2d07b4ce3b622a66f14c93bdc: stopping tablet replica
I20240501 14:10:42.466228 26887 raft_consensus.cc:2229] T bb291a0c743c4e638f79da36cc77faa3 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:42.466540 26887 raft_consensus.cc:2258] T bb291a0c743c4e638f79da36cc77faa3 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:42.470715 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:42.470965 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:42.471766 26887 ts_tablet_manager.cc:1875] T bb291a0c743c4e638f79da36cc77faa3 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:42.475577 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:42.477636 26887 ts_tablet_manager.cc:1888] T bb291a0c743c4e638f79da36cc77faa3 P fb1e6bc2d07b4ce3b622a66f14c93bdc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:10:42.477728 26887 log.cc:1196] T bb291a0c743c4e638f79da36cc77faa3 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal/wals/bb291a0c743c4e638f79da36cc77faa3
I20240501 14:10:42.478345 19218 catalog_manager.cc:4830] TS fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847): tablet bb291a0c743c4e638f79da36cc77faa3 (table rebalance_test_table_5 [id=4d7621338643426d90fe03eb71654cb8]) successfully deleted
I20240501 14:10:42.479250 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:42.495885 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:42.511281 21247 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:42.531214 19617 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:10:42.584399 26882 tablet_bootstrap.cc:492] T 814ba2247a314a4797aa0f9f624aaeb6 P 43ce4a4c954f4a5f80954b38933c189d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:42.584759 26882 tablet_bootstrap.cc:492] T 814ba2247a314a4797aa0f9f624aaeb6 P 43ce4a4c954f4a5f80954b38933c189d: Bootstrap complete.
I20240501 14:10:42.584946 26882 ts_tablet_manager.cc:1367] T 814ba2247a314a4797aa0f9f624aaeb6 P 43ce4a4c954f4a5f80954b38933c189d: Time spent bootstrapping tablet: real 0.129s user 0.030s sys 0.000s
I20240501 14:10:42.585402 26882 raft_consensus.cc:348] T 814ba2247a314a4797aa0f9f624aaeb6 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:42.585517 26882 raft_consensus.cc:729] T 814ba2247a314a4797aa0f9f624aaeb6 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43ce4a4c954f4a5f80954b38933c189d, State: Initialized, Role: FOLLOWER
I20240501 14:10:42.585678 26882 consensus_queue.cc:260] T 814ba2247a314a4797aa0f9f624aaeb6 P 43ce4a4c954f4a5f80954b38933c189d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:42.585938 26882 ts_tablet_manager.cc:1398] T 814ba2247a314a4797aa0f9f624aaeb6 P 43ce4a4c954f4a5f80954b38933c189d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:42.586576 19396 tablet_copy_service.cc:342] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Request end of tablet copy session 43ce4a4c954f4a5f80954b38933c189d-814ba2247a314a4797aa0f9f624aaeb6 received from {username='slave'} at 127.7.39.135:49533
I20240501 14:10:42.586690 19396 tablet_copy_service.cc:433] P fb1e6bc2d07b4ce3b622a66f14c93bdc: ending tablet copy session 43ce4a4c954f4a5f80954b38933c189d-814ba2247a314a4797aa0f9f624aaeb6 on tablet 814ba2247a314a4797aa0f9f624aaeb6 with peer 43ce4a4c954f4a5f80954b38933c189d
I20240501 14:10:42.605099 21268 consensus_queue.cc:237] T 12ffa74f7a5b472aac57253693273cad P e6246b183293412494919f1e7efc9448 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:42.609907 19375 raft_consensus.cc:1261] T 12ffa74f7a5b472aac57253693273cad P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 2 FOLLOWER]: Refusing update from remote peer e6246b183293412494919f1e7efc9448: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:10:42.610623 25974 consensus_queue.cc:1035] T 12ffa74f7a5b472aac57253693273cad P e6246b183293412494919f1e7efc9448 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:10:42.610756 19767 raft_consensus.cc:1261] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 [term 2 FOLLOWER]: Refusing update from remote peer e6246b183293412494919f1e7efc9448: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:10:42.611186 25974 consensus_queue.cc:1035] T 12ffa74f7a5b472aac57253693273cad P e6246b183293412494919f1e7efc9448 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:10:42.616003 25974 raft_consensus.cc:2932] T 12ffa74f7a5b472aac57253693273cad P e6246b183293412494919f1e7efc9448 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:42.620049 19232 catalog_manager.cc:5484] T 12ffa74f7a5b472aac57253693273cad P e6246b183293412494919f1e7efc9448 reported cstate change: config changed from index 4 to 6, VOTER ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131) evicted. New cstate: current_term: 2 leader_uuid: "e6246b183293412494919f1e7efc9448" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:42.620851 19375 raft_consensus.cc:2932] T 12ffa74f7a5b472aac57253693273cad P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:42.618847 19767 raft_consensus.cc:2932] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:42.632021 19617 tablet_service.cc:1514] Processing DeleteTablet for tablet 12ffa74f7a5b472aac57253693273cad with delete_type TABLET_DATA_TOMBSTONED (TS ec8cde17a00645e7b38d5d29f7425367 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47238
I20240501 14:10:42.651978 26992 tablet_replica.cc:330] T 12ffa74f7a5b472aac57253693273cad P ec8cde17a00645e7b38d5d29f7425367: stopping tablet replica
I20240501 14:10:42.653270 26992 raft_consensus.cc:2229] T 12ffa74f7a5b472aac57253693273cad P ec8cde17a00645e7b38d5d29f7425367 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:42.655505 26992 raft_consensus.cc:2258] T 12ffa74f7a5b472aac57253693273cad P ec8cde17a00645e7b38d5d29f7425367 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:42.656231 26992 ts_tablet_manager.cc:1875] T 12ffa74f7a5b472aac57253693273cad P ec8cde17a00645e7b38d5d29f7425367: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:42.661996 19637 consensus_queue.cc:237] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:42.663601 21268 raft_consensus.cc:1261] T cc8e888d46d34a22b6d4409ec2ab53a9 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:10:42.663662 21398 raft_consensus.cc:1261] T cc8e888d46d34a22b6d4409ec2ab53a9 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:10:42.664050 25959 consensus_queue.cc:1035] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:10:42.664551 25959 consensus_queue.cc:1035] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:10:42.664594 19375 raft_consensus.cc:1261] T cc8e888d46d34a22b6d4409ec2ab53a9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:10:42.665807 25959 consensus_queue.cc:1035] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:10:42.671290 27020 raft_consensus.cc:2932] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:42.672190 21268 raft_consensus.cc:2932] T cc8e888d46d34a22b6d4409ec2ab53a9 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:42.659009 26992 ts_tablet_manager.cc:1888] T 12ffa74f7a5b472aac57253693273cad P ec8cde17a00645e7b38d5d29f7425367: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 14:10:42.675724 26992 log.cc:1196] T 12ffa74f7a5b472aac57253693273cad P ec8cde17a00645e7b38d5d29f7425367: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal/wals/12ffa74f7a5b472aac57253693273cad
I20240501 14:10:42.676921 19219 catalog_manager.cc:4830] TS ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201): tablet 12ffa74f7a5b472aac57253693273cad (table rebalance_test_table_5 [id=4d7621338643426d90fe03eb71654cb8]) successfully deleted
I20240501 14:10:42.678771 19375 raft_consensus.cc:2932] T cc8e888d46d34a22b6d4409ec2ab53a9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:42.679339 19233 catalog_manager.cc:5484] T cc8e888d46d34a22b6d4409ec2ab53a9 P e6246b183293412494919f1e7efc9448 reported cstate change: config changed from index 5 to 6, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:42.680380 21398 raft_consensus.cc:2932] T cc8e888d46d34a22b6d4409ec2ab53a9 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:42.683171 19485 tablet_service.cc:1514] Processing DeleteTablet for tablet cc8e888d46d34a22b6d4409ec2ab53a9 with delete_type TABLET_DATA_TOMBSTONED (TS 99a0be6ef2144b7fb247ba9d3f051d66 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37600
I20240501 14:10:42.690148 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:42.693921 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:42.706514 21247 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:42.707156 19617 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:10:42.707242 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:42.710995 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:42.733822 27041 tablet_replica.cc:330] T cc8e888d46d34a22b6d4409ec2ab53a9 P 99a0be6ef2144b7fb247ba9d3f051d66: stopping tablet replica
I20240501 14:10:42.733989 27041 raft_consensus.cc:2229] T cc8e888d46d34a22b6d4409ec2ab53a9 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:42.734148 27041 raft_consensus.cc:2258] T cc8e888d46d34a22b6d4409ec2ab53a9 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:42.734784 27041 ts_tablet_manager.cc:1875] T cc8e888d46d34a22b6d4409ec2ab53a9 P 99a0be6ef2144b7fb247ba9d3f051d66: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:42.742506 27041 ts_tablet_manager.cc:1888] T cc8e888d46d34a22b6d4409ec2ab53a9 P 99a0be6ef2144b7fb247ba9d3f051d66: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:10:42.742611 27041 log.cc:1196] T cc8e888d46d34a22b6d4409ec2ab53a9 P 99a0be6ef2144b7fb247ba9d3f051d66: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal/wals/cc8e888d46d34a22b6d4409ec2ab53a9
I20240501 14:10:42.743444 19219 catalog_manager.cc:4830] TS 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829): tablet cc8e888d46d34a22b6d4409ec2ab53a9 (table rebalance_test_table_7 [id=ed1a7ed87fa9480ba18a052130ecdc05]) successfully deleted
I20240501 14:10:42.750629 21492 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:42.766043 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:42.767799 21492 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:42.769539 21247 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:42.771072 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:42.783382 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:42.793740 19617 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:10:42.796882 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:42.798153 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:42.838641 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:42.866871 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:42.871461 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:42.874428 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:42.877106 21247 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:42.882555 21492 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:42.892357 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:42.893234 19617 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:10:42.916208 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:42.922395 19506 consensus_queue.cc:237] T f6ca088ce3aa4b97aa54c8ce6d700921 P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:42.923678 21398 raft_consensus.cc:1261] T f6ca088ce3aa4b97aa54c8ce6d700921 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Refusing update from remote peer 99a0be6ef2144b7fb247ba9d3f051d66: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:10:42.924149 25626 consensus_queue.cc:1035] T f6ca088ce3aa4b97aa54c8ce6d700921 P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:10:42.929572 25626 raft_consensus.cc:2932] T f6ca088ce3aa4b97aa54c8ce6d700921 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:42.932572 19233 catalog_manager.cc:5484] T f6ca088ce3aa4b97aa54c8ce6d700921 P 99a0be6ef2144b7fb247ba9d3f051d66 reported cstate change: config changed from index 5 to 6, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New cstate: current_term: 1 leader_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:42.933990 21398 raft_consensus.cc:2932] T f6ca088ce3aa4b97aa54c8ce6d700921 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:42.935397 19748 tablet_service.cc:1514] Processing DeleteTablet for tablet f6ca088ce3aa4b97aa54c8ce6d700921 with delete_type TABLET_DATA_TOMBSTONED (TS da3c4e948ff747678f18456c134d7ed4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36530
I20240501 14:10:42.937350 27143 tablet_replica.cc:330] T f6ca088ce3aa4b97aa54c8ce6d700921 P da3c4e948ff747678f18456c134d7ed4: stopping tablet replica
I20240501 14:10:42.937410 19506 consensus_queue.cc:237] T f6ca088ce3aa4b97aa54c8ce6d700921 P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:10:42.937547 27143 raft_consensus.cc:2229] T f6ca088ce3aa4b97aa54c8ce6d700921 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:42.937719 27143 raft_consensus.cc:2258] T f6ca088ce3aa4b97aa54c8ce6d700921 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:42.938308 27143 ts_tablet_manager.cc:1875] T f6ca088ce3aa4b97aa54c8ce6d700921 P da3c4e948ff747678f18456c134d7ed4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 14:10:42.939131 19440 consensus_peers.cc:485] T f6ca088ce3aa4b97aa54c8ce6d700921 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136:34895): Couldn't send request to peer 778c78105092428bbcde75ab2cdb6a05. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f6ca088ce3aa4b97aa54c8ce6d700921. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:42.939849 21398 raft_consensus.cc:1261] T f6ca088ce3aa4b97aa54c8ce6d700921 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Refusing update from remote peer 99a0be6ef2144b7fb247ba9d3f051d66: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:10:42.940275 25626 consensus_queue.cc:1035] T f6ca088ce3aa4b97aa54c8ce6d700921 P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:10:42.948047 25626 raft_consensus.cc:2932] T f6ca088ce3aa4b97aa54c8ce6d700921 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:42.949882 19219 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet f6ca088ce3aa4b97aa54c8ce6d700921 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:10:42.950620 21398 raft_consensus.cc:2932] T f6ca088ce3aa4b97aa54c8ce6d700921 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:42.950888 19233 catalog_manager.cc:5484] T f6ca088ce3aa4b97aa54c8ce6d700921 P 99a0be6ef2144b7fb247ba9d3f051d66 reported cstate change: config changed from index 6 to 7, VOTER 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136) added. New cstate: current_term: 1 leader_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:42.953658 21398 raft_consensus.cc:1203] T 814ba2247a314a4797aa0f9f624aaeb6 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:10:42.954511 27143 ts_tablet_manager.cc:1888] T f6ca088ce3aa4b97aa54c8ce6d700921 P da3c4e948ff747678f18456c134d7ed4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:10:42.954636 27143 log.cc:1196] T f6ca088ce3aa4b97aa54c8ce6d700921 P da3c4e948ff747678f18456c134d7ed4: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal/wals/f6ca088ce3aa4b97aa54c8ce6d700921
I20240501 14:10:42.956373 19220 catalog_manager.cc:4830] TS da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357): tablet f6ca088ce3aa4b97aa54c8ce6d700921 (table rebalance_test_table_8 [id=3866c35dd0e64d87b6049e5e14cb0d31]) successfully deleted
I20240501 14:10:43.004856 19637 consensus_queue.cc:237] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:43.006181 19375 raft_consensus.cc:1261] T ffb4b6a2c19b497f8117fbbc9ea50082 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:43.006696 27020 consensus_queue.cc:1035] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:43.006868 21268 raft_consensus.cc:1261] T ffb4b6a2c19b497f8117fbbc9ea50082 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:43.008065 27020 consensus_queue.cc:1035] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:43.010836 27021 raft_consensus.cc:2932] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:43.013994 19375 raft_consensus.cc:2932] T ffb4b6a2c19b497f8117fbbc9ea50082 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:43.015254 19234 catalog_manager.cc:5484] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 reported cstate change: config changed from index 4 to 5, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:43.018414 21268 raft_consensus.cc:2932] T ffb4b6a2c19b497f8117fbbc9ea50082 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:43.020488 19485 tablet_service.cc:1514] Processing DeleteTablet for tablet ffb4b6a2c19b497f8117fbbc9ea50082 with delete_type TABLET_DATA_TOMBSTONED (TS 99a0be6ef2144b7fb247ba9d3f051d66 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:37600
I20240501 14:10:43.020792 27041 tablet_replica.cc:330] T ffb4b6a2c19b497f8117fbbc9ea50082 P 99a0be6ef2144b7fb247ba9d3f051d66: stopping tablet replica
I20240501 14:10:43.020951 27041 raft_consensus.cc:2229] T ffb4b6a2c19b497f8117fbbc9ea50082 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:43.021121 27041 raft_consensus.cc:2258] T ffb4b6a2c19b497f8117fbbc9ea50082 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:43.021776 27041 ts_tablet_manager.cc:1875] T ffb4b6a2c19b497f8117fbbc9ea50082 P 99a0be6ef2144b7fb247ba9d3f051d66: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:43.024724 27041 ts_tablet_manager.cc:1888] T ffb4b6a2c19b497f8117fbbc9ea50082 P 99a0be6ef2144b7fb247ba9d3f051d66: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:10:43.024806 27041 log.cc:1196] T ffb4b6a2c19b497f8117fbbc9ea50082 P 99a0be6ef2144b7fb247ba9d3f051d66: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal/wals/ffb4b6a2c19b497f8117fbbc9ea50082
I20240501 14:10:43.025739 19219 catalog_manager.cc:4830] TS 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829): tablet ffb4b6a2c19b497f8117fbbc9ea50082 (table rebalance_test_table_3 [id=35d6f8fac5394197b987aa44dab206c1]) successfully deleted
I20240501 14:10:43.051910 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:43.071527 19617 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:10:43.075502 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:43.093240 21247 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:43.096305 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:43.147188 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:43.161031 21492 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:43.162014 21247 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:43.162236 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:43.162292 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:43.170393 19617 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:10:43.171265 21247 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:43.176121 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:43.177376 21492 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:43.184545 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:43.190150 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:43.211983 19617 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:10:43.221266 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:43.225910 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:43.230249 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:43.230691 21492 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:43.231653 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:43.237691 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:43.241691 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:43.340221 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:43.342134 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:43.349092 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:43.361191 21492 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:43.369719 21247 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:43.373651 19637 consensus_queue.cc:237] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:43.375102 21268 raft_consensus.cc:1261] T db0f01ee72b24ec9b84410cf94ef9759 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:10:43.375522 27020 consensus_queue.cc:1035] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:10:43.375774 21398 raft_consensus.cc:1261] T db0f01ee72b24ec9b84410cf94ef9759 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:10:43.376149 27020 consensus_queue.cc:1035] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:10:43.386763 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:43.391952 19617 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:10:43.396099 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:43.418278 21268 raft_consensus.cc:2932] T db0f01ee72b24ec9b84410cf94ef9759 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:43.420344 27020 raft_consensus.cc:2932] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:43.421584 19232 catalog_manager.cc:5484] T db0f01ee72b24ec9b84410cf94ef9759 P e6246b183293412494919f1e7efc9448 reported cstate change: config changed from index 6 to 7, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) evicted. New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:43.425415 21398 raft_consensus.cc:2932] T db0f01ee72b24ec9b84410cf94ef9759 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:43.429577 19356 tablet_service.cc:1514] Processing DeleteTablet for tablet db0f01ee72b24ec9b84410cf94ef9759 with delete_type TABLET_DATA_TOMBSTONED (TS fb1e6bc2d07b4ce3b622a66f14c93bdc not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:46838
I20240501 14:10:43.444626 27443 tablet_replica.cc:330] T db0f01ee72b24ec9b84410cf94ef9759 P fb1e6bc2d07b4ce3b622a66f14c93bdc: stopping tablet replica
I20240501 14:10:43.444998 27443 raft_consensus.cc:2229] T db0f01ee72b24ec9b84410cf94ef9759 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:43.445712 27443 raft_consensus.cc:2258] T db0f01ee72b24ec9b84410cf94ef9759 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:43.446367 27443 ts_tablet_manager.cc:1875] T db0f01ee72b24ec9b84410cf94ef9759 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:43.449674 27443 ts_tablet_manager.cc:1888] T db0f01ee72b24ec9b84410cf94ef9759 P fb1e6bc2d07b4ce3b622a66f14c93bdc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:10:43.449857 27443 log.cc:1196] T db0f01ee72b24ec9b84410cf94ef9759 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal/wals/db0f01ee72b24ec9b84410cf94ef9759
I20240501 14:10:43.451860 19218 catalog_manager.cc:4830] TS fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847): tablet db0f01ee72b24ec9b84410cf94ef9759 (table rebalance_test_table_2 [id=6f6b0a378a484e5889d733ff982d9eb7]) successfully deleted
I20240501 14:10:43.458824 19636 consensus_queue.cc:237] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:43.460333 21398 raft_consensus.cc:1261] T cc8e888d46d34a22b6d4409ec2ab53a9 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:10:43.460436 21268 raft_consensus.cc:1261] T cc8e888d46d34a22b6d4409ec2ab53a9 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:10:43.460764 27020 consensus_queue.cc:1035] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:10:43.461047 27021 consensus_queue.cc:1035] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:10:43.463232 27439 raft_consensus.cc:2932] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:43.463825 21268 raft_consensus.cc:2932] T cc8e888d46d34a22b6d4409ec2ab53a9 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:43.463831 21398 raft_consensus.cc:2932] T cc8e888d46d34a22b6d4409ec2ab53a9 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:43.467377 19232 catalog_manager.cc:5484] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 reported cstate change: config changed from index 6 to 7, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) evicted. New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:43.470664 19356 tablet_service.cc:1514] Processing DeleteTablet for tablet cc8e888d46d34a22b6d4409ec2ab53a9 with delete_type TABLET_DATA_TOMBSTONED (TS fb1e6bc2d07b4ce3b622a66f14c93bdc not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:46838
I20240501 14:10:43.470912 27443 tablet_replica.cc:330] T cc8e888d46d34a22b6d4409ec2ab53a9 P fb1e6bc2d07b4ce3b622a66f14c93bdc: stopping tablet replica
I20240501 14:10:43.471246 27443 raft_consensus.cc:2229] T cc8e888d46d34a22b6d4409ec2ab53a9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:43.471475 27443 raft_consensus.cc:2258] T cc8e888d46d34a22b6d4409ec2ab53a9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:43.473538 27443 ts_tablet_manager.cc:1875] T cc8e888d46d34a22b6d4409ec2ab53a9 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:43.477113 27443 ts_tablet_manager.cc:1888] T cc8e888d46d34a22b6d4409ec2ab53a9 P fb1e6bc2d07b4ce3b622a66f14c93bdc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:10:43.477207 27443 log.cc:1196] T cc8e888d46d34a22b6d4409ec2ab53a9 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal/wals/cc8e888d46d34a22b6d4409ec2ab53a9
I20240501 14:10:43.477818 19218 catalog_manager.cc:4830] TS fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847): tablet cc8e888d46d34a22b6d4409ec2ab53a9 (table rebalance_test_table_7 [id=ed1a7ed87fa9480ba18a052130ecdc05]) successfully deleted
I20240501 14:10:43.489861 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:43.494390 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:43.503998 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:43.508023 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:43.509079 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:43.509799 19617 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:10:43.516728 21492 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:43.549983 27515 ts_tablet_manager.cc:902] T f6ca088ce3aa4b97aa54c8ce6d700921 P 778c78105092428bbcde75ab2cdb6a05: Initiating tablet copy from peer 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829)
I20240501 14:10:43.550462 27515 tablet_copy_client.cc:308] T f6ca088ce3aa4b97aa54c8ce6d700921 P 778c78105092428bbcde75ab2cdb6a05: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.130:41829
I20240501 14:10:43.550923 19527 tablet_copy_service.cc:140] P 99a0be6ef2144b7fb247ba9d3f051d66: Received BeginTabletCopySession request for tablet f6ca088ce3aa4b97aa54c8ce6d700921 from peer 778c78105092428bbcde75ab2cdb6a05 ({username='slave'} at 127.7.39.136:39441)
I20240501 14:10:43.551038 19527 tablet_copy_service.cc:161] P 99a0be6ef2144b7fb247ba9d3f051d66: Beginning new tablet copy session on tablet f6ca088ce3aa4b97aa54c8ce6d700921 from peer 778c78105092428bbcde75ab2cdb6a05 at {username='slave'} at 127.7.39.136:39441: session id = 778c78105092428bbcde75ab2cdb6a05-f6ca088ce3aa4b97aa54c8ce6d700921
I20240501 14:10:43.552136 19527 tablet_copy_source_session.cc:205] T f6ca088ce3aa4b97aa54c8ce6d700921 P 99a0be6ef2144b7fb247ba9d3f051d66: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:43.553061 27515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6ca088ce3aa4b97aa54c8ce6d700921. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:43.555964 27515 tablet_copy_client.cc:783] T f6ca088ce3aa4b97aa54c8ce6d700921 P 778c78105092428bbcde75ab2cdb6a05: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:43.556121 27515 tablet_copy_client.cc:647] T f6ca088ce3aa4b97aa54c8ce6d700921 P 778c78105092428bbcde75ab2cdb6a05: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:43.562952 21247 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:43.568467 19505 consensus_queue.cc:237] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:10:43.570168 21522 raft_consensus.cc:1261] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Refusing update from remote peer 99a0be6ef2144b7fb247ba9d3f051d66: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:43.570199 19636 raft_consensus.cc:1261] T 272eee3450cf4f48a4e1de2dcb28c0f7 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Refusing update from remote peer 99a0be6ef2144b7fb247ba9d3f051d66: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:43.570639 27515 tablet_copy_client.cc:523] T f6ca088ce3aa4b97aa54c8ce6d700921 P 778c78105092428bbcde75ab2cdb6a05: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:43.571542 27152 consensus_queue.cc:1035] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:43.572201 27515 tablet_bootstrap.cc:492] T f6ca088ce3aa4b97aa54c8ce6d700921 P 778c78105092428bbcde75ab2cdb6a05: Bootstrap starting.
I20240501 14:10:43.573525 27152 raft_consensus.cc:2932] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:43.573854 21522 raft_consensus.cc:2932] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:43.575965 27152 consensus_queue.cc:1035] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:43.577126 19225 catalog_manager.cc:5484] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 reported cstate change: config changed from index 4 to 5, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New cstate: current_term: 1 leader_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:43.577215 19636 raft_consensus.cc:2932] T 272eee3450cf4f48a4e1de2dcb28c0f7 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:43.579919 19748 tablet_service.cc:1514] Processing DeleteTablet for tablet 272eee3450cf4f48a4e1de2dcb28c0f7 with delete_type TABLET_DATA_TOMBSTONED (TS da3c4e948ff747678f18456c134d7ed4 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:36530
I20240501 14:10:43.581333 27551 tablet_replica.cc:330] T 272eee3450cf4f48a4e1de2dcb28c0f7 P da3c4e948ff747678f18456c134d7ed4: stopping tablet replica
I20240501 14:10:43.581530 27551 raft_consensus.cc:2229] T 272eee3450cf4f48a4e1de2dcb28c0f7 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:43.581707 27551 raft_consensus.cc:2258] T 272eee3450cf4f48a4e1de2dcb28c0f7 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:43.583536 27551 ts_tablet_manager.cc:1875] T 272eee3450cf4f48a4e1de2dcb28c0f7 P da3c4e948ff747678f18456c134d7ed4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:43.586143 27551 ts_tablet_manager.cc:1888] T 272eee3450cf4f48a4e1de2dcb28c0f7 P da3c4e948ff747678f18456c134d7ed4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:10:43.589062 27551 log.cc:1196] T 272eee3450cf4f48a4e1de2dcb28c0f7 P da3c4e948ff747678f18456c134d7ed4: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal/wals/272eee3450cf4f48a4e1de2dcb28c0f7
I20240501 14:10:43.595968 19220 catalog_manager.cc:4830] TS da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357): tablet 272eee3450cf4f48a4e1de2dcb28c0f7 (table rebalance_test_table_7 [id=ed1a7ed87fa9480ba18a052130ecdc05]) successfully deleted
I20240501 14:10:43.608877 27515 tablet_bootstrap.cc:492] T f6ca088ce3aa4b97aa54c8ce6d700921 P 778c78105092428bbcde75ab2cdb6a05: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:43.609366 27515 tablet_bootstrap.cc:492] T f6ca088ce3aa4b97aa54c8ce6d700921 P 778c78105092428bbcde75ab2cdb6a05: Bootstrap complete.
I20240501 14:10:43.609675 27515 ts_tablet_manager.cc:1367] T f6ca088ce3aa4b97aa54c8ce6d700921 P 778c78105092428bbcde75ab2cdb6a05: Time spent bootstrapping tablet: real 0.038s user 0.025s sys 0.004s
I20240501 14:10:43.610270 27515 raft_consensus.cc:348] T f6ca088ce3aa4b97aa54c8ce6d700921 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:10:43.610575 27515 raft_consensus.cc:729] T f6ca088ce3aa4b97aa54c8ce6d700921 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 778c78105092428bbcde75ab2cdb6a05, State: Initialized, Role: FOLLOWER
I20240501 14:10:43.610817 27515 consensus_queue.cc:260] T f6ca088ce3aa4b97aa54c8ce6d700921 P 778c78105092428bbcde75ab2cdb6a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:10:43.611280 27515 ts_tablet_manager.cc:1398] T f6ca088ce3aa4b97aa54c8ce6d700921 P 778c78105092428bbcde75ab2cdb6a05: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:10:43.611848 19527 tablet_copy_service.cc:342] P 99a0be6ef2144b7fb247ba9d3f051d66: Request end of tablet copy session 778c78105092428bbcde75ab2cdb6a05-f6ca088ce3aa4b97aa54c8ce6d700921 received from {username='slave'} at 127.7.39.136:39441
I20240501 14:10:43.611929 19527 tablet_copy_service.cc:433] P 99a0be6ef2144b7fb247ba9d3f051d66: ending tablet copy session 778c78105092428bbcde75ab2cdb6a05-f6ca088ce3aa4b97aa54c8ce6d700921 on tablet f6ca088ce3aa4b97aa54c8ce6d700921 with peer 778c78105092428bbcde75ab2cdb6a05
I20240501 14:10:43.628621 19375 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "814ba2247a314a4797aa0f9f624aaeb6"
dest_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34022
I20240501 14:10:43.628818 19375 raft_consensus.cc:595] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Received request to transfer leadership
I20240501 14:10:43.663378 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:43.676185 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:43.685472 21492 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:43.697191 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:43.697896 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:43.700546 21247 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:43.701427 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:43.721463 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:43.721463 21492 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:43.722592 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:43.729097 19617 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:10:43.737632 19617 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:10:43.747830 19356 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:43.754729 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:43.767501 21247 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:43.772069 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:43.856282 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:43.864746 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:43.871227 26252 raft_consensus.cc:979] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc: : Instructing follower e6246b183293412494919f1e7efc9448 to start an election
I20240501 14:10:43.871320 26252 raft_consensus.cc:1067] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Signalling peer e6246b183293412494919f1e7efc9448 to start an election
I20240501 14:10:43.872875 21268 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "814ba2247a314a4797aa0f9f624aaeb6"
dest_uuid: "e6246b183293412494919f1e7efc9448"
from {username='slave'} at 127.7.39.129:44111
I20240501 14:10:43.873018 21268 raft_consensus.cc:482] T 814ba2247a314a4797aa0f9f624aaeb6 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:10:43.873075 21268 raft_consensus.cc:3037] T 814ba2247a314a4797aa0f9f624aaeb6 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:43.874104 21268 raft_consensus.cc:504] T 814ba2247a314a4797aa0f9f624aaeb6 P e6246b183293412494919f1e7efc9448 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:43.874938 19375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "814ba2247a314a4797aa0f9f624aaeb6" candidate_uuid: "e6246b183293412494919f1e7efc9448" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc"
I20240501 14:10:43.875089 21268 leader_election.cc:288] T 814ba2247a314a4797aa0f9f624aaeb6 P e6246b183293412494919f1e7efc9448 [CANDIDATE]: Term 2 election: Requested vote from peers fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847), 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135:38751)
I20240501 14:10:43.875155 19375 raft_consensus.cc:3032] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:10:43.875314 19375 raft_consensus.cc:729] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fb1e6bc2d07b4ce3b622a66f14c93bdc, State: Running, Role: LEADER
I20240501 14:10:43.875559 19375 consensus_queue.cc:260] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:43.875905 19375 raft_consensus.cc:3037] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:43.877012 19375 raft_consensus.cc:2454] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6246b183293412494919f1e7efc9448 in term 2.
I20240501 14:10:43.877475 21201 leader_election.cc:302] T 814ba2247a314a4797aa0f9f624aaeb6 P e6246b183293412494919f1e7efc9448 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6246b183293412494919f1e7efc9448, fb1e6bc2d07b4ce3b622a66f14c93bdc; no voters:
I20240501 14:10:43.877640 26986 raft_consensus.cc:2781] T 814ba2247a314a4797aa0f9f624aaeb6 P e6246b183293412494919f1e7efc9448 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:10:43.877766 26986 raft_consensus.cc:686] T 814ba2247a314a4797aa0f9f624aaeb6 P e6246b183293412494919f1e7efc9448 [term 2 LEADER]: Becoming Leader. State: Replica: e6246b183293412494919f1e7efc9448, State: Running, Role: LEADER
I20240501 14:10:43.877903 26986 consensus_queue.cc:237] T 814ba2247a314a4797aa0f9f624aaeb6 P e6246b183293412494919f1e7efc9448 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:43.880889 21492 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:43.881052 19232 catalog_manager.cc:5484] T 814ba2247a314a4797aa0f9f624aaeb6 P e6246b183293412494919f1e7efc9448 reported cstate change: term changed from 1 to 2, leader changed from fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) to e6246b183293412494919f1e7efc9448 (127.7.39.134). New cstate: current_term: 2 leader_uuid: "e6246b183293412494919f1e7efc9448" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:43.882424 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:43.889747 21398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "814ba2247a314a4797aa0f9f624aaeb6" candidate_uuid: "e6246b183293412494919f1e7efc9448" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "43ce4a4c954f4a5f80954b38933c189d"
I20240501 14:10:43.889999 21398 raft_consensus.cc:3037] T 814ba2247a314a4797aa0f9f624aaeb6 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:43.891294 21398 raft_consensus.cc:2454] T 814ba2247a314a4797aa0f9f624aaeb6 P 43ce4a4c954f4a5f80954b38933c189d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6246b183293412494919f1e7efc9448 in term 2.
I20240501 14:10:43.899498 21247 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:43.899521 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:43.912065 19356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:43.924602 21247 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:43.930253 19617 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:10:43.932157 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:43.937415 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:43.953177 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:43.958038 21492 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:43.961177 19617 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:10:43.961793 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:43.995662 19356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:44.024583 19375 consensus_queue.cc:237] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:44.027282 21268 raft_consensus.cc:1261] T d63afd081986473aa2715d074b123a93 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:10:44.027746 27423 consensus_queue.cc:1035] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:10:44.029847 26252 raft_consensus.cc:2932] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:44.030752 21268 raft_consensus.cc:2932] T d63afd081986473aa2715d074b123a93 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:44.031940 19375 consensus_queue.cc:237] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:10:44.033285 19232 catalog_manager.cc:5484] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc reported cstate change: config changed from index 5 to 6, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New cstate: current_term: 1 leader_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:44.040060 21522 raft_consensus.cc:1203] T f6ca088ce3aa4b97aa54c8ce6d700921 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:10:44.040506 19505 raft_consensus.cc:1261] T 410a7dd0dd734684a82875912966657d P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:44.040110 21530 raft_consensus.cc:1261] T 410a7dd0dd734684a82875912966657d P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:44.042169 27423 consensus_queue.cc:1035] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:44.042374 26252 consensus_queue.cc:1035] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:44.044435 19748 tablet_service.cc:1514] Processing DeleteTablet for tablet d63afd081986473aa2715d074b123a93 with delete_type TABLET_DATA_TOMBSTONED (TS da3c4e948ff747678f18456c134d7ed4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36530
I20240501 14:10:44.044706 27551 tablet_replica.cc:330] T d63afd081986473aa2715d074b123a93 P da3c4e948ff747678f18456c134d7ed4: stopping tablet replica
I20240501 14:10:44.044870 27551 raft_consensus.cc:2229] T d63afd081986473aa2715d074b123a93 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:44.045091 27551 raft_consensus.cc:2258] T d63afd081986473aa2715d074b123a93 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:44.045483 19375 consensus_queue.cc:237] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:44.045845 27551 ts_tablet_manager.cc:1875] T d63afd081986473aa2715d074b123a93 P da3c4e948ff747678f18456c134d7ed4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:44.047097 21268 raft_consensus.cc:1261] T d63afd081986473aa2715d074b123a93 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:10:44.047492 27423 consensus_queue.cc:1035] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 14:10:44.047988 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829): Couldn't send request to peer 99a0be6ef2144b7fb247ba9d3f051d66. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:44.048871 27551 ts_tablet_manager.cc:1888] T d63afd081986473aa2715d074b123a93 P da3c4e948ff747678f18456c134d7ed4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:10:44.048966 27551 log.cc:1196] T d63afd081986473aa2715d074b123a93 P da3c4e948ff747678f18456c134d7ed4: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal/wals/d63afd081986473aa2715d074b123a93
I20240501 14:10:44.049324 27423 raft_consensus.cc:2932] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } }
I20240501 14:10:44.049497 19220 catalog_manager.cc:4830] TS da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357): tablet d63afd081986473aa2715d074b123a93 (table rebalance_test_table_6 [id=afc36d383e3e4795ad11804e9bebcb51]) successfully deleted
I20240501 14:10:44.051455 19218 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet d63afd081986473aa2715d074b123a93 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:10:44.051789 21268 raft_consensus.cc:2932] T d63afd081986473aa2715d074b123a93 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } }
I20240501 14:10:44.054145 26252 raft_consensus.cc:2932] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:44.054591 19231 catalog_manager.cc:5484] T d63afd081986473aa2715d074b123a93 P e6246b183293412494919f1e7efc9448 reported cstate change: config changed from index 6 to 7, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) added. New cstate: current_term: 1 leader_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } }
I20240501 14:10:44.054706 21530 raft_consensus.cc:2932] T 410a7dd0dd734684a82875912966657d P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:44.058084 19225 catalog_manager.cc:5484] T 410a7dd0dd734684a82875912966657d P 778c78105092428bbcde75ab2cdb6a05 reported cstate change: config changed from index 4 to 5, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New cstate: current_term: 1 leader_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:44.060792 19748 tablet_service.cc:1514] Processing DeleteTablet for tablet 410a7dd0dd734684a82875912966657d with delete_type TABLET_DATA_TOMBSTONED (TS da3c4e948ff747678f18456c134d7ed4 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:36530
I20240501 14:10:44.061723 27551 tablet_replica.cc:330] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4: stopping tablet replica
I20240501 14:10:44.061873 27551 raft_consensus.cc:2229] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:44.062047 27551 raft_consensus.cc:2258] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:44.062737 27551 ts_tablet_manager.cc:1875] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:44.063405 19505 raft_consensus.cc:2932] T 410a7dd0dd734684a82875912966657d P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:44.066462 27551 ts_tablet_manager.cc:1888] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:10:44.066581 27551 log.cc:1196] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal/wals/410a7dd0dd734684a82875912966657d
I20240501 14:10:44.067152 19220 catalog_manager.cc:4830] TS da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357): tablet 410a7dd0dd734684a82875912966657d (table rebalance_test_table_2 [id=6f6b0a378a484e5889d733ff982d9eb7]) successfully deleted
I20240501 14:10:44.080802 21492 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:44.085834 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:44.087697 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:44.090821 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:44.093554 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:44.097630 21247 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:44.124107 19617 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:10:44.128269 19356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:44.156726 19375 consensus_queue.cc:237] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:44.159067 21136 raft_consensus.cc:1261] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:10:44.159118 21398 raft_consensus.cc:1261] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:10:44.159574 27786 consensus_queue.cc:1035] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:10:44.159978 26252 consensus_queue.cc:1035] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:10:44.160279 19505 raft_consensus.cc:1261] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:10:44.161569 26252 consensus_queue.cc:1035] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:10:44.168671 27786 raft_consensus.cc:2932] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:44.169126 19505 raft_consensus.cc:2932] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:44.172948 21136 raft_consensus.cc:2932] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:44.175233 19232 catalog_manager.cc:5484] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc reported cstate change: config changed from index 5 to 6, VOTER ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131) evicted. New cstate: current_term: 1 leader_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:44.181071 19617 tablet_service.cc:1514] Processing DeleteTablet for tablet 98af8b06b9324e3fa7cdb898b7b9fab9 with delete_type TABLET_DATA_TOMBSTONED (TS ec8cde17a00645e7b38d5d29f7425367 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47238
I20240501 14:10:44.192350 27888 tablet_replica.cc:330] T 98af8b06b9324e3fa7cdb898b7b9fab9 P ec8cde17a00645e7b38d5d29f7425367: stopping tablet replica
I20240501 14:10:44.192636 27888 raft_consensus.cc:2229] T 98af8b06b9324e3fa7cdb898b7b9fab9 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:44.193226 27888 raft_consensus.cc:2258] T 98af8b06b9324e3fa7cdb898b7b9fab9 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:44.193971 27888 ts_tablet_manager.cc:1875] T 98af8b06b9324e3fa7cdb898b7b9fab9 P ec8cde17a00645e7b38d5d29f7425367: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:44.207267 27888 ts_tablet_manager.cc:1888] T 98af8b06b9324e3fa7cdb898b7b9fab9 P ec8cde17a00645e7b38d5d29f7425367: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:10:44.209617 21398 raft_consensus.cc:2932] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:44.210309 27888 log.cc:1196] T 98af8b06b9324e3fa7cdb898b7b9fab9 P ec8cde17a00645e7b38d5d29f7425367: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal/wals/98af8b06b9324e3fa7cdb898b7b9fab9
I20240501 14:10:44.211470 19219 catalog_manager.cc:4830] TS ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201): tablet 98af8b06b9324e3fa7cdb898b7b9fab9 (table rebalance_test_table_5 [id=4d7621338643426d90fe03eb71654cb8]) successfully deleted
I20240501 14:10:44.234736 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:44.237381 21492 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:44.237576 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:44.246042 21247 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:44.254127 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:44.270906 19617 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:10:44.282709 19356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:44.300326 21398 raft_consensus.cc:1261] T 814ba2247a314a4797aa0f9f624aaeb6 P 43ce4a4c954f4a5f80954b38933c189d [term 2 FOLLOWER]: Refusing update from remote peer e6246b183293412494919f1e7efc9448: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 14:10:44.300791 27734 consensus_queue.cc:1035] T 814ba2247a314a4797aa0f9f624aaeb6 P e6246b183293412494919f1e7efc9448 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:10:44.304999 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:44.309165 19375 raft_consensus.cc:1261] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 2 FOLLOWER]: Refusing update from remote peer e6246b183293412494919f1e7efc9448: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 14:10:44.310410 26986 consensus_queue.cc:1035] T 814ba2247a314a4797aa0f9f624aaeb6 P e6246b183293412494919f1e7efc9448 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:10:44.325170 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:44.326156 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:44.332219 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:44.334477 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:44.351893 19617 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:10:44.372690 21247 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:44.373193 21492 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:44.386111 19356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:44.441615 19375 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f69020bb6c244848a6b0a50fd2b4f3eb"
dest_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34210
I20240501 14:10:44.441972 19375 raft_consensus.cc:595] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Received request to transfer leadership
I20240501 14:10:44.444171 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:44.449812 19767 consensus_queue.cc:237] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:44.451819 21136 raft_consensus.cc:1261] T 1d408b075d784953a57554bcdf19158a P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:44.452303 19505 raft_consensus.cc:1261] T 1d408b075d784953a57554bcdf19158a P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:44.453650 25934 consensus_queue.cc:1035] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:44.453974 25934 consensus_queue.cc:1035] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:44.454697 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:44.459329 19617 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:10:44.460566 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:44.462138 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:44.462620 21247 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:44.467645 28061 ts_tablet_manager.cc:902] T d63afd081986473aa2715d074b123a93 P 99a0be6ef2144b7fb247ba9d3f051d66: Initiating tablet copy from peer fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847)
I20240501 14:10:44.468504 28061 tablet_copy_client.cc:308] T d63afd081986473aa2715d074b123a93 P 99a0be6ef2144b7fb247ba9d3f051d66: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.129:35847
I20240501 14:10:44.469367 19396 tablet_copy_service.cc:140] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Received BeginTabletCopySession request for tablet d63afd081986473aa2715d074b123a93 from peer 99a0be6ef2144b7fb247ba9d3f051d66 ({username='slave'} at 127.7.39.130:45871)
I20240501 14:10:44.469485 19396 tablet_copy_service.cc:161] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Beginning new tablet copy session on tablet d63afd081986473aa2715d074b123a93 from peer 99a0be6ef2144b7fb247ba9d3f051d66 at {username='slave'} at 127.7.39.130:45871: session id = 99a0be6ef2144b7fb247ba9d3f051d66-d63afd081986473aa2715d074b123a93
I20240501 14:10:44.470428 19396 tablet_copy_source_session.cc:205] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:44.471529 28061 ts_tablet_manager.cc:1875] T d63afd081986473aa2715d074b123a93 P 99a0be6ef2144b7fb247ba9d3f051d66: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:10:44.473903 28061 ts_tablet_manager.cc:1888] T d63afd081986473aa2715d074b123a93 P 99a0be6ef2144b7fb247ba9d3f051d66: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240501 14:10:44.474073 28061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d63afd081986473aa2715d074b123a93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:44.476219 28061 tablet_copy_client.cc:783] T d63afd081986473aa2715d074b123a93 P 99a0be6ef2144b7fb247ba9d3f051d66: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:44.476449 28061 tablet_copy_client.cc:647] T d63afd081986473aa2715d074b123a93 P 99a0be6ef2144b7fb247ba9d3f051d66: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:44.477861 28061 tablet_copy_client.cc:523] T d63afd081986473aa2715d074b123a93 P 99a0be6ef2144b7fb247ba9d3f051d66: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:44.479646 28061 tablet_bootstrap.cc:492] T d63afd081986473aa2715d074b123a93 P 99a0be6ef2144b7fb247ba9d3f051d66: Bootstrap starting.
I20240501 14:10:44.489212 21492 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:44.499369 19356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:44.505249 28053 raft_consensus.cc:2932] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:44.505828 19505 raft_consensus.cc:2932] T 1d408b075d784953a57554bcdf19158a P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:44.509014 21136 raft_consensus.cc:2932] T 1d408b075d784953a57554bcdf19158a P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:44.509027 19229 catalog_manager.cc:5484] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4 reported cstate change: config changed from index 4 to 5, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) evicted. New cstate: current_term: 1 leader_uuid: "da3c4e948ff747678f18456c134d7ed4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:44.512742 19356 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d408b075d784953a57554bcdf19158a with delete_type TABLET_DATA_TOMBSTONED (TS fb1e6bc2d07b4ce3b622a66f14c93bdc not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:46838
I20240501 14:10:44.541829 28061 tablet_bootstrap.cc:492] T d63afd081986473aa2715d074b123a93 P 99a0be6ef2144b7fb247ba9d3f051d66: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:44.542286 28061 tablet_bootstrap.cc:492] T d63afd081986473aa2715d074b123a93 P 99a0be6ef2144b7fb247ba9d3f051d66: Bootstrap complete.
I20240501 14:10:44.542479 28061 ts_tablet_manager.cc:1367] T d63afd081986473aa2715d074b123a93 P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent bootstrapping tablet: real 0.063s user 0.029s sys 0.000s
I20240501 14:10:44.542963 28061 raft_consensus.cc:348] T d63afd081986473aa2715d074b123a93 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:44.543138 28061 raft_consensus.cc:729] T d63afd081986473aa2715d074b123a93 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a0be6ef2144b7fb247ba9d3f051d66, State: Initialized, Role: FOLLOWER
I20240501 14:10:44.543290 28061 consensus_queue.cc:260] T d63afd081986473aa2715d074b123a93 P 99a0be6ef2144b7fb247ba9d3f051d66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:44.544246 28061 ts_tablet_manager.cc:1398] T d63afd081986473aa2715d074b123a93 P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:10:44.546094 19396 tablet_copy_service.cc:342] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Request end of tablet copy session 99a0be6ef2144b7fb247ba9d3f051d66-d63afd081986473aa2715d074b123a93 received from {username='slave'} at 127.7.39.130:45871
I20240501 14:10:44.546175 19396 tablet_copy_service.cc:433] P fb1e6bc2d07b4ce3b622a66f14c93bdc: ending tablet copy session 99a0be6ef2144b7fb247ba9d3f051d66-d63afd081986473aa2715d074b123a93 on tablet d63afd081986473aa2715d074b123a93 with peer 99a0be6ef2144b7fb247ba9d3f051d66
I20240501 14:10:44.575420 28071 tablet_replica.cc:330] T 1d408b075d784953a57554bcdf19158a P fb1e6bc2d07b4ce3b622a66f14c93bdc: stopping tablet replica
I20240501 14:10:44.575672 28071 raft_consensus.cc:2229] T 1d408b075d784953a57554bcdf19158a P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:44.575928 28071 raft_consensus.cc:2258] T 1d408b075d784953a57554bcdf19158a P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:44.576517 28071 ts_tablet_manager.cc:1875] T 1d408b075d784953a57554bcdf19158a P fb1e6bc2d07b4ce3b622a66f14c93bdc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:44.583842 28071 ts_tablet_manager.cc:1888] T 1d408b075d784953a57554bcdf19158a P fb1e6bc2d07b4ce3b622a66f14c93bdc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:10:44.584022 28071 log.cc:1196] T 1d408b075d784953a57554bcdf19158a P fb1e6bc2d07b4ce3b622a66f14c93bdc: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal/wals/1d408b075d784953a57554bcdf19158a
I20240501 14:10:44.585183 19218 catalog_manager.cc:4830] TS fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847): tablet 1d408b075d784953a57554bcdf19158a (table rebalance_test_table_0 [id=b52269ef7c494f74a5e4f0d5cb3b8a47]) successfully deleted
I20240501 14:10:44.626009 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:44.633812 21492 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:44.635301 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:44.640796 21247 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:44.641122 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:44.644162 19356 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:44.650216 19617 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:10:44.657332 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:44.690165 21378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:44.701670 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:44.710883 19617 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:10:44.712174 21247 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:44.715198 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:44.717509 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:44.720631 26252 raft_consensus.cc:979] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc: : Instructing follower 43ce4a4c954f4a5f80954b38933c189d to start an election
I20240501 14:10:44.720715 26252 raft_consensus.cc:1067] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Signalling peer 43ce4a4c954f4a5f80954b38933c189d to start an election
I20240501 14:10:44.721139 21398 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f69020bb6c244848a6b0a50fd2b4f3eb"
dest_uuid: "43ce4a4c954f4a5f80954b38933c189d"
from {username='slave'} at 127.7.39.129:58141
I20240501 14:10:44.721320 21398 raft_consensus.cc:482] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:10:44.721374 21398 raft_consensus.cc:3037] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:44.722414 21398 raft_consensus.cc:504] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:44.723425 21492 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:44.724969 21398 leader_election.cc:288] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d [CANDIDATE]: Term 2 election: Requested vote from peers fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847), 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133:46883)
I20240501 14:10:44.725425 19375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69020bb6c244848a6b0a50fd2b4f3eb" candidate_uuid: "43ce4a4c954f4a5f80954b38933c189d" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc"
I20240501 14:10:44.725557 19375 raft_consensus.cc:3032] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:10:44.725611 19375 raft_consensus.cc:729] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fb1e6bc2d07b4ce3b622a66f14c93bdc, State: Running, Role: LEADER
I20240501 14:10:44.725754 19375 consensus_queue.cc:260] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:44.725896 19375 raft_consensus.cc:3037] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:44.726914 19375 raft_consensus.cc:2454] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43ce4a4c954f4a5f80954b38933c189d in term 2.
I20240501 14:10:44.727838 21332 leader_election.cc:302] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43ce4a4c954f4a5f80954b38933c189d, fb1e6bc2d07b4ce3b622a66f14c93bdc; no voters:
I20240501 14:10:44.728055 27909 raft_consensus.cc:2781] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:10:44.728211 27909 raft_consensus.cc:686] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d [term 2 LEADER]: Becoming Leader. State: Replica: 43ce4a4c954f4a5f80954b38933c189d, State: Running, Role: LEADER
I20240501 14:10:44.728493 27909 consensus_queue.cc:237] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:44.731886 19227 catalog_manager.cc:5484] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d reported cstate change: term changed from 1 to 2, leader changed from fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) to 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135). New cstate: current_term: 2 leader_uuid: "43ce4a4c954f4a5f80954b38933c189d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:44.743343 19375 consensus_queue.cc:237] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:44.746606 21398 raft_consensus.cc:1261] T ad7c255baee74e629e2976a942a4174f P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:10:44.748010 27786 consensus_queue.cc:1035] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:10:44.749534 19356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:44.754384 21136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69020bb6c244848a6b0a50fd2b4f3eb" candidate_uuid: "43ce4a4c954f4a5f80954b38933c189d" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "5bf81f3d986e428190d3f7727f36d96c"
I20240501 14:10:44.754597 21136 raft_consensus.cc:3037] T f69020bb6c244848a6b0a50fd2b4f3eb P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:44.757076 21136 raft_consensus.cc:2454] T f69020bb6c244848a6b0a50fd2b4f3eb P 5bf81f3d986e428190d3f7727f36d96c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43ce4a4c954f4a5f80954b38933c189d in term 2.
I20240501 14:10:44.769340 27786 raft_consensus.cc:2932] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:44.771589 21398 raft_consensus.cc:2932] T ad7c255baee74e629e2976a942a4174f P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:44.772459 19227 catalog_manager.cc:5484] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc reported cstate change: config changed from index 7 to 8, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New cstate: current_term: 1 leader_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:44.775764 19748 tablet_service.cc:1514] Processing DeleteTablet for tablet ad7c255baee74e629e2976a942a4174f with delete_type TABLET_DATA_TOMBSTONED (TS da3c4e948ff747678f18456c134d7ed4 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:36530
I20240501 14:10:44.776206 28230 tablet_replica.cc:330] T ad7c255baee74e629e2976a942a4174f P da3c4e948ff747678f18456c134d7ed4: stopping tablet replica
I20240501 14:10:44.776432 28230 raft_consensus.cc:2229] T ad7c255baee74e629e2976a942a4174f P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:44.776583 19375 consensus_queue.cc:237] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:44.776793 28230 raft_consensus.cc:2258] T ad7c255baee74e629e2976a942a4174f P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:44.777438 28230 ts_tablet_manager.cc:1875] T ad7c255baee74e629e2976a942a4174f P da3c4e948ff747678f18456c134d7ed4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 14:10:44.778473 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ad7c255baee74e629e2976a942a4174f. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:44.778766 21398 raft_consensus.cc:1261] T ad7c255baee74e629e2976a942a4174f P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:10:44.779193 27786 consensus_queue.cc:1035] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:10:44.780429 28230 ts_tablet_manager.cc:1888] T ad7c255baee74e629e2976a942a4174f P da3c4e948ff747678f18456c134d7ed4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 14:10:44.780532 28230 log.cc:1196] T ad7c255baee74e629e2976a942a4174f P da3c4e948ff747678f18456c134d7ed4: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal/wals/ad7c255baee74e629e2976a942a4174f
I20240501 14:10:44.781087 19220 catalog_manager.cc:4830] TS da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357): tablet ad7c255baee74e629e2976a942a4174f (table rebalance_test_table_6 [id=afc36d383e3e4795ad11804e9bebcb51]) successfully deleted
I20240501 14:10:44.781023 27786 raft_consensus.cc:2932] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:44.782483 19218 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet ad7c255baee74e629e2976a942a4174f with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:10:44.782888 21398 raft_consensus.cc:2932] T ad7c255baee74e629e2976a942a4174f P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:44.785779 19229 catalog_manager.cc:5484] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc reported cstate change: config changed from index 8 to 9, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New cstate: current_term: 1 leader_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:44.837453 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:44.841533 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:44.847776 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:44.848763 21247 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:44.864570 19617 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:10:44.865357 19356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:44.875530 19375 consensus_queue.cc:237] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:10:44.876855 21530 raft_consensus.cc:1261] T 410a7dd0dd734684a82875912966657d P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:10:44.877400 28229 consensus_queue.cc:1035] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:10:44.880187 28229 raft_consensus.cc:2932] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:44.883165 21530 raft_consensus.cc:2932] T 410a7dd0dd734684a82875912966657d P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:44.883671 19229 catalog_manager.cc:5484] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc reported cstate change: config changed from index 5 to 6, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New cstate: current_term: 1 leader_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:44.892109 19375 consensus_queue.cc:237] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:44.892997 19485 tablet_service.cc:1514] Processing DeleteTablet for tablet 410a7dd0dd734684a82875912966657d with delete_type TABLET_DATA_TOMBSTONED (TS 99a0be6ef2144b7fb247ba9d3f051d66 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37600
W20240501 14:10:44.894168 19311 consensus_peers.cc:485] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357): Couldn't send request to peer da3c4e948ff747678f18456c134d7ed4. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:44.894443 28338 tablet_replica.cc:330] T 410a7dd0dd734684a82875912966657d P 99a0be6ef2144b7fb247ba9d3f051d66: stopping tablet replica
I20240501 14:10:44.894745 28338 raft_consensus.cc:2229] T 410a7dd0dd734684a82875912966657d P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:44.894973 28338 raft_consensus.cc:2258] T 410a7dd0dd734684a82875912966657d P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:44.895995 28338 ts_tablet_manager.cc:1875] T 410a7dd0dd734684a82875912966657d P 99a0be6ef2144b7fb247ba9d3f051d66: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:44.898347 28338 ts_tablet_manager.cc:1888] T 410a7dd0dd734684a82875912966657d P 99a0be6ef2144b7fb247ba9d3f051d66: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:10:44.898475 28338 log.cc:1196] T 410a7dd0dd734684a82875912966657d P 99a0be6ef2144b7fb247ba9d3f051d66: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal/wals/410a7dd0dd734684a82875912966657d
I20240501 14:10:44.899314 21530 raft_consensus.cc:1261] T 410a7dd0dd734684a82875912966657d P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:10:44.899916 26252 consensus_queue.cc:1035] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:10:44.900075 19219 catalog_manager.cc:4830] TS 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829): tablet 410a7dd0dd734684a82875912966657d (table rebalance_test_table_2 [id=6f6b0a378a484e5889d733ff982d9eb7]) successfully deleted
I20240501 14:10:44.901866 28229 raft_consensus.cc:2932] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } }
I20240501 14:10:44.902333 21530 raft_consensus.cc:2932] T 410a7dd0dd734684a82875912966657d P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } }
I20240501 14:10:44.903008 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:44.903769 19218 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 410a7dd0dd734684a82875912966657d with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:10:44.905397 19229 catalog_manager.cc:5484] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc reported cstate change: config changed from index 6 to 7, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) added. New cstate: current_term: 1 leader_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } }
I20240501 14:10:44.911375 21492 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:44.947500 19505 raft_consensus.cc:1203] T d63afd081986473aa2715d074b123a93 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:10:45.008813 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:45.017496 19617 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:10:45.019337 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:45.027797 21247 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:45.038240 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:45.045459 19356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:45.064198 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:45.066423 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:45.067339 21492 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:45.072139 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:45.077049 21247 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:45.077292 19617 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:10:45.083289 19356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:45.086248 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:45.091770 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:45.099196 21492 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:45.151754 21136 raft_consensus.cc:1261] T f69020bb6c244848a6b0a50fd2b4f3eb P 5bf81f3d986e428190d3f7727f36d96c [term 2 FOLLOWER]: Refusing update from remote peer 43ce4a4c954f4a5f80954b38933c189d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 14:10:45.152271 27909 consensus_queue.cc:1035] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:10:45.161144 19375 raft_consensus.cc:1261] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 2 FOLLOWER]: Refusing update from remote peer 43ce4a4c954f4a5f80954b38933c189d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 14:10:45.161687 28214 consensus_queue.cc:1035] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:10:45.171391 28502 ts_tablet_manager.cc:902] T ad7c255baee74e629e2976a942a4174f P e6246b183293412494919f1e7efc9448: Initiating tablet copy from peer fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847)
I20240501 14:10:45.171839 28502 tablet_copy_client.cc:308] T ad7c255baee74e629e2976a942a4174f P e6246b183293412494919f1e7efc9448: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.129:35847
I20240501 14:10:45.172542 19396 tablet_copy_service.cc:140] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Received BeginTabletCopySession request for tablet ad7c255baee74e629e2976a942a4174f from peer e6246b183293412494919f1e7efc9448 ({username='slave'} at 127.7.39.134:34039)
I20240501 14:10:45.172649 19396 tablet_copy_service.cc:161] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Beginning new tablet copy session on tablet ad7c255baee74e629e2976a942a4174f from peer e6246b183293412494919f1e7efc9448 at {username='slave'} at 127.7.39.134:34039: session id = e6246b183293412494919f1e7efc9448-ad7c255baee74e629e2976a942a4174f
I20240501 14:10:45.173635 19396 tablet_copy_source_session.cc:205] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:45.174824 28502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad7c255baee74e629e2976a942a4174f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:45.187355 28502 tablet_copy_client.cc:783] T ad7c255baee74e629e2976a942a4174f P e6246b183293412494919f1e7efc9448: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:45.188867 28502 tablet_copy_client.cc:647] T ad7c255baee74e629e2976a942a4174f P e6246b183293412494919f1e7efc9448: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:45.212123 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:45.212682 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:45.213922 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:45.216785 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:45.217042 21492 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:45.225723 19617 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:10:45.242017 21247 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:45.243683 28502 tablet_copy_client.cc:523] T ad7c255baee74e629e2976a942a4174f P e6246b183293412494919f1e7efc9448: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:45.245405 28502 tablet_bootstrap.cc:492] T ad7c255baee74e629e2976a942a4174f P e6246b183293412494919f1e7efc9448: Bootstrap starting.
I20240501 14:10:45.263599 19356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:45.269557 19767 consensus_queue.cc:237] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:45.270797 21136 raft_consensus.cc:1261] T 1d408b075d784953a57554bcdf19158a P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:10:45.271222 28053 consensus_queue.cc:1035] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:10:45.273135 28053 raft_consensus.cc:2932] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:45.276319 19227 catalog_manager.cc:5484] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4 reported cstate change: config changed from index 5 to 6, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New cstate: current_term: 1 leader_uuid: "da3c4e948ff747678f18456c134d7ed4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:45.276404 21136 raft_consensus.cc:2932] T 1d408b075d784953a57554bcdf19158a P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:45.281673 21268 consensus_queue.cc:237] T 814ba2247a314a4797aa0f9f624aaeb6 P e6246b183293412494919f1e7efc9448 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:45.282989 19485 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d408b075d784953a57554bcdf19158a with delete_type TABLET_DATA_TOMBSTONED (TS 99a0be6ef2144b7fb247ba9d3f051d66 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37600
I20240501 14:10:45.283247 28338 tablet_replica.cc:330] T 1d408b075d784953a57554bcdf19158a P 99a0be6ef2144b7fb247ba9d3f051d66: stopping tablet replica
I20240501 14:10:45.283403 28338 raft_consensus.cc:2229] T 1d408b075d784953a57554bcdf19158a P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:45.283569 28338 raft_consensus.cc:2258] T 1d408b075d784953a57554bcdf19158a P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:45.283900 19767 consensus_queue.cc:237] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:45.284140 21398 raft_consensus.cc:1261] T 814ba2247a314a4797aa0f9f624aaeb6 P 43ce4a4c954f4a5f80954b38933c189d [term 2 FOLLOWER]: Refusing update from remote peer e6246b183293412494919f1e7efc9448: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:10:45.284384 28338 ts_tablet_manager.cc:1875] T 1d408b075d784953a57554bcdf19158a P 99a0be6ef2144b7fb247ba9d3f051d66: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:45.284600 28325 consensus_queue.cc:1035] T 814ba2247a314a4797aa0f9f624aaeb6 P e6246b183293412494919f1e7efc9448 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:10:45.285306 21136 raft_consensus.cc:1261] T 1d408b075d784953a57554bcdf19158a P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:10:45.285702 25934 consensus_queue.cc:1035] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:10:45.286789 28338 ts_tablet_manager.cc:1888] T 1d408b075d784953a57554bcdf19158a P 99a0be6ef2144b7fb247ba9d3f051d66: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
W20240501 14:10:45.286819 19703 consensus_peers.cc:485] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4 -> Peer 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829): Couldn't send request to peer 99a0be6ef2144b7fb247ba9d3f051d66. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:45.286850 28338 log.cc:1196] T 1d408b075d784953a57554bcdf19158a P 99a0be6ef2144b7fb247ba9d3f051d66: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal/wals/1d408b075d784953a57554bcdf19158a
I20240501 14:10:45.287804 19219 catalog_manager.cc:4830] TS 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829): tablet 1d408b075d784953a57554bcdf19158a (table rebalance_test_table_0 [id=b52269ef7c494f74a5e4f0d5cb3b8a47]) successfully deleted
I20240501 14:10:45.287940 28325 raft_consensus.cc:2932] T 814ba2247a314a4797aa0f9f624aaeb6 P e6246b183293412494919f1e7efc9448 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:45.287946 25934 raft_consensus.cc:2932] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } }
I20240501 14:10:45.290359 19220 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 1d408b075d784953a57554bcdf19158a with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:10:45.290355 21398 raft_consensus.cc:2932] T 814ba2247a314a4797aa0f9f624aaeb6 P 43ce4a4c954f4a5f80954b38933c189d [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:45.290505 21136 raft_consensus.cc:2932] T 1d408b075d784953a57554bcdf19158a P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } }
I20240501 14:10:45.290683 19226 catalog_manager.cc:5484] T 814ba2247a314a4797aa0f9f624aaeb6 P e6246b183293412494919f1e7efc9448 reported cstate change: config changed from index 7 to 9, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) evicted. New cstate: current_term: 2 leader_uuid: "e6246b183293412494919f1e7efc9448" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:45.290992 19227 catalog_manager.cc:5484] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4 reported cstate change: config changed from index 6 to 7, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) added. New cstate: current_term: 1 leader_uuid: "da3c4e948ff747678f18456c134d7ed4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } }
I20240501 14:10:45.296252 19356 tablet_service.cc:1514] Processing DeleteTablet for tablet 814ba2247a314a4797aa0f9f624aaeb6 with delete_type TABLET_DATA_TOMBSTONED (TS fb1e6bc2d07b4ce3b622a66f14c93bdc not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:46838
I20240501 14:10:45.297523 28579 tablet_replica.cc:330] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc: stopping tablet replica
I20240501 14:10:45.297817 28579 raft_consensus.cc:2229] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:45.297997 28579 raft_consensus.cc:2258] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:45.298800 28579 ts_tablet_manager.cc:1875] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:45.302198 28579 ts_tablet_manager.cc:1888] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.8
I20240501 14:10:45.302345 28579 log.cc:1196] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal/wals/814ba2247a314a4797aa0f9f624aaeb6
I20240501 14:10:45.303200 19218 catalog_manager.cc:4830] TS fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847): tablet 814ba2247a314a4797aa0f9f624aaeb6 (table rebalance_test_table_4 [id=151dfad0ef40470ab9c5203c42400b60]) successfully deleted
I20240501 14:10:45.306641 21268 consensus_queue.cc:237] T 814ba2247a314a4797aa0f9f624aaeb6 P e6246b183293412494919f1e7efc9448 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:45.309376 21398 raft_consensus.cc:1261] T 814ba2247a314a4797aa0f9f624aaeb6 P 43ce4a4c954f4a5f80954b38933c189d [term 2 FOLLOWER]: Refusing update from remote peer e6246b183293412494919f1e7efc9448: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 14:10:45.309795 28325 consensus_queue.cc:1035] T 814ba2247a314a4797aa0f9f624aaeb6 P e6246b183293412494919f1e7efc9448 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:10:45.312469 28325 raft_consensus.cc:2932] T 814ba2247a314a4797aa0f9f624aaeb6 P e6246b183293412494919f1e7efc9448 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:45.314980 19219 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 814ba2247a314a4797aa0f9f624aaeb6 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:10:45.315364 21398 raft_consensus.cc:2932] T 814ba2247a314a4797aa0f9f624aaeb6 P 43ce4a4c954f4a5f80954b38933c189d [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:45.315598 19232 catalog_manager.cc:5484] T 814ba2247a314a4797aa0f9f624aaeb6 P e6246b183293412494919f1e7efc9448 reported cstate change: config changed from index 9 to 10, VOTER 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133) added. New cstate: current_term: 2 leader_uuid: "e6246b183293412494919f1e7efc9448" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
W20240501 14:10:45.325701 21204 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P e6246b183293412494919f1e7efc9448 -> Peer 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133:46883): Couldn't send request to peer 5bf81f3d986e428190d3f7727f36d96c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 814ba2247a314a4797aa0f9f624aaeb6. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:45.338773 28502 tablet_bootstrap.cc:492] T ad7c255baee74e629e2976a942a4174f P e6246b183293412494919f1e7efc9448: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:45.339407 28502 tablet_bootstrap.cc:492] T ad7c255baee74e629e2976a942a4174f P e6246b183293412494919f1e7efc9448: Bootstrap complete.
I20240501 14:10:45.339624 28502 ts_tablet_manager.cc:1367] T ad7c255baee74e629e2976a942a4174f P e6246b183293412494919f1e7efc9448: Time spent bootstrapping tablet: real 0.094s user 0.032s sys 0.001s
I20240501 14:10:45.340042 28502 raft_consensus.cc:348] T ad7c255baee74e629e2976a942a4174f P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:45.340162 28502 raft_consensus.cc:729] T ad7c255baee74e629e2976a942a4174f P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6246b183293412494919f1e7efc9448, State: Initialized, Role: FOLLOWER
I20240501 14:10:45.340325 28502 consensus_queue.cc:260] T ad7c255baee74e629e2976a942a4174f P e6246b183293412494919f1e7efc9448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:45.341060 28502 ts_tablet_manager.cc:1398] T ad7c255baee74e629e2976a942a4174f P e6246b183293412494919f1e7efc9448: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:10:45.341881 19396 tablet_copy_service.cc:342] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Request end of tablet copy session e6246b183293412494919f1e7efc9448-ad7c255baee74e629e2976a942a4174f received from {username='slave'} at 127.7.39.134:34039
I20240501 14:10:45.341972 19396 tablet_copy_service.cc:433] P fb1e6bc2d07b4ce3b622a66f14c93bdc: ending tablet copy session e6246b183293412494919f1e7efc9448-ad7c255baee74e629e2976a942a4174f on tablet ad7c255baee74e629e2976a942a4174f with peer e6246b183293412494919f1e7efc9448
I20240501 14:10:45.365324 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:45.365265 19767 consensus_queue.cc:237] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:45.366799 21268 raft_consensus.cc:1261] T 11f64b59f83e458f8cf51fd5bcd36fee P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:10:45.367309 28573 consensus_queue.cc:1035] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:10:45.370306 28573 raft_consensus.cc:2932] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:45.371811 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:45.373356 21268 raft_consensus.cc:2932] T 11f64b59f83e458f8cf51fd5bcd36fee P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:45.376389 19231 catalog_manager.cc:5484] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 reported cstate change: config changed from index 5 to 6, VOTER ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131) evicted. New cstate: current_term: 1 leader_uuid: "da3c4e948ff747678f18456c134d7ed4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:45.379339 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:45.382540 19617 tablet_service.cc:1514] Processing DeleteTablet for tablet 11f64b59f83e458f8cf51fd5bcd36fee with delete_type TABLET_DATA_TOMBSTONED (TS ec8cde17a00645e7b38d5d29f7425367 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47238
I20240501 14:10:45.384153 19767 consensus_queue.cc:237] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
W20240501 14:10:45.385686 19705 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133:46883): Couldn't send request to peer 5bf81f3d986e428190d3f7727f36d96c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 11f64b59f83e458f8cf51fd5bcd36fee. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:45.386138 21268 raft_consensus.cc:1261] T 11f64b59f83e458f8cf51fd5bcd36fee P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:10:45.386528 25934 consensus_queue.cc:1035] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:10:45.390380 21247 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:45.390503 25934 raft_consensus.cc:2932] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:45.393453 19220 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 11f64b59f83e458f8cf51fd5bcd36fee with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:10:45.393640 21268 raft_consensus.cc:2932] T 11f64b59f83e458f8cf51fd5bcd36fee P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:45.398102 19229 catalog_manager.cc:5484] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 reported cstate change: config changed from index 6 to 7, VOTER 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133) added. New cstate: current_term: 1 leader_uuid: "da3c4e948ff747678f18456c134d7ed4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:45.400269 19356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:45.402390 19617 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:10:45.404794 28663 tablet_replica.cc:330] T 11f64b59f83e458f8cf51fd5bcd36fee P ec8cde17a00645e7b38d5d29f7425367: stopping tablet replica
I20240501 14:10:45.405014 28663 raft_consensus.cc:2229] T 11f64b59f83e458f8cf51fd5bcd36fee P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:45.405243 28663 raft_consensus.cc:2258] T 11f64b59f83e458f8cf51fd5bcd36fee P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:45.405968 28663 ts_tablet_manager.cc:1875] T 11f64b59f83e458f8cf51fd5bcd36fee P ec8cde17a00645e7b38d5d29f7425367: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:45.406256 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:45.411437 28663 ts_tablet_manager.cc:1888] T 11f64b59f83e458f8cf51fd5bcd36fee P ec8cde17a00645e7b38d5d29f7425367: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:10:45.411599 28663 log.cc:1196] T 11f64b59f83e458f8cf51fd5bcd36fee P ec8cde17a00645e7b38d5d29f7425367: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal/wals/11f64b59f83e458f8cf51fd5bcd36fee
I20240501 14:10:45.414626 19219 catalog_manager.cc:4830] TS ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201): tablet 11f64b59f83e458f8cf51fd5bcd36fee (table rebalance_test_table_8 [id=3866c35dd0e64d87b6049e5e14cb0d31]) successfully deleted
I20240501 14:10:45.451202 21492 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:45.498263 28755 ts_tablet_manager.cc:902] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4: Initiating tablet copy from peer fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847)
I20240501 14:10:45.499701 28755 tablet_copy_client.cc:308] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.129:35847
I20240501 14:10:45.501300 19396 tablet_copy_service.cc:140] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Received BeginTabletCopySession request for tablet 410a7dd0dd734684a82875912966657d from peer da3c4e948ff747678f18456c134d7ed4 ({username='slave'} at 127.7.39.132:46133)
I20240501 14:10:45.501623 19396 tablet_copy_service.cc:161] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Beginning new tablet copy session on tablet 410a7dd0dd734684a82875912966657d from peer da3c4e948ff747678f18456c134d7ed4 at {username='slave'} at 127.7.39.132:46133: session id = da3c4e948ff747678f18456c134d7ed4-410a7dd0dd734684a82875912966657d
I20240501 14:10:45.502914 19396 tablet_copy_source_session.cc:205] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:45.504053 28755 ts_tablet_manager.cc:1875] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:10:45.506443 28755 ts_tablet_manager.cc:1888] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240501 14:10:45.506654 28755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 410a7dd0dd734684a82875912966657d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:45.508409 28755 tablet_copy_client.cc:783] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:45.508575 28755 tablet_copy_client.cc:647] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:45.511075 28755 tablet_copy_client.cc:523] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:45.512738 28755 tablet_bootstrap.cc:492] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4: Bootstrap starting.
I20240501 14:10:45.555771 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:45.559835 28755 tablet_bootstrap.cc:492] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:45.560351 28755 tablet_bootstrap.cc:492] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4: Bootstrap complete.
I20240501 14:10:45.560623 28755 ts_tablet_manager.cc:1367] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4: Time spent bootstrapping tablet: real 0.048s user 0.032s sys 0.001s
I20240501 14:10:45.561293 28755 raft_consensus.cc:348] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:45.561470 28755 raft_consensus.cc:729] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3c4e948ff747678f18456c134d7ed4, State: Initialized, Role: FOLLOWER
I20240501 14:10:45.561655 28755 consensus_queue.cc:260] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:45.562484 28755 ts_tablet_manager.cc:1398] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:10:45.564057 19396 tablet_copy_service.cc:342] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Request end of tablet copy session da3c4e948ff747678f18456c134d7ed4-410a7dd0dd734684a82875912966657d received from {username='slave'} at 127.7.39.132:46133
I20240501 14:10:45.564147 19396 tablet_copy_service.cc:433] P fb1e6bc2d07b4ce3b622a66f14c93bdc: ending tablet copy session da3c4e948ff747678f18456c134d7ed4-410a7dd0dd734684a82875912966657d on tablet 410a7dd0dd734684a82875912966657d with peer da3c4e948ff747678f18456c134d7ed4
I20240501 14:10:45.565366 19356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:45.578653 21247 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:45.582460 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:45.590643 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:45.592923 19617 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:10:45.601855 21492 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:45.611863 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:45.620924 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:45.622414 21268 raft_consensus.cc:1203] T ad7c255baee74e629e2976a942a4174f P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240501 14:10:45.640390 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:45.648803 21247 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:45.652640 19617 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:10:45.657780 19356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:45.659287 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:45.669526 19506 consensus_queue.cc:237] T 55830424a98348e382759c54afdeabcd P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:10:45.671209 21530 raft_consensus.cc:1261] T 55830424a98348e382759c54afdeabcd P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Refusing update from remote peer 99a0be6ef2144b7fb247ba9d3f051d66: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:10:45.672118 27152 consensus_queue.cc:1035] T 55830424a98348e382759c54afdeabcd P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:10:45.679244 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:45.682570 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:45.686442 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:45.687197 19617 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:10:45.695477 27152 raft_consensus.cc:2932] T 55830424a98348e382759c54afdeabcd P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:45.698257 21530 raft_consensus.cc:2932] T 55830424a98348e382759c54afdeabcd P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:45.700663 19230 catalog_manager.cc:5484] T 55830424a98348e382759c54afdeabcd P 99a0be6ef2144b7fb247ba9d3f051d66 reported cstate change: config changed from index 5 to 6, VOTER ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131) evicted. New cstate: current_term: 1 leader_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:45.701445 19356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:45.707595 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:45.711124 19617 tablet_service.cc:1514] Processing DeleteTablet for tablet 55830424a98348e382759c54afdeabcd with delete_type TABLET_DATA_TOMBSTONED (TS ec8cde17a00645e7b38d5d29f7425367 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47238
I20240501 14:10:45.715163 28663 tablet_replica.cc:330] T 55830424a98348e382759c54afdeabcd P ec8cde17a00645e7b38d5d29f7425367: stopping tablet replica
I20240501 14:10:45.715317 28663 raft_consensus.cc:2229] T 55830424a98348e382759c54afdeabcd P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:45.715468 28663 raft_consensus.cc:2258] T 55830424a98348e382759c54afdeabcd P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:45.716104 28663 ts_tablet_manager.cc:1875] T 55830424a98348e382759c54afdeabcd P ec8cde17a00645e7b38d5d29f7425367: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:45.718276 28663 ts_tablet_manager.cc:1888] T 55830424a98348e382759c54afdeabcd P ec8cde17a00645e7b38d5d29f7425367: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:10:45.718359 28663 log.cc:1196] T 55830424a98348e382759c54afdeabcd P ec8cde17a00645e7b38d5d29f7425367: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal/wals/55830424a98348e382759c54afdeabcd
I20240501 14:10:45.718931 19219 catalog_manager.cc:4830] TS ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201): tablet 55830424a98348e382759c54afdeabcd (table rebalance_test_table_0 [id=b52269ef7c494f74a5e4f0d5cb3b8a47]) successfully deleted
I20240501 14:10:45.720131 19506 consensus_queue.cc:237] T 55830424a98348e382759c54afdeabcd P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:45.720438 21492 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:45.723712 21492 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:45.725394 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:45.733590 21530 raft_consensus.cc:1261] T 55830424a98348e382759c54afdeabcd P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Refusing update from remote peer 99a0be6ef2144b7fb247ba9d3f051d66: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:10:45.734102 27152 consensus_queue.cc:1035] T 55830424a98348e382759c54afdeabcd P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:10:45.741792 21247 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:45.768646 19443 consensus_peers.cc:485] T 55830424a98348e382759c54afdeabcd P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133:46883): Couldn't send request to peer 5bf81f3d986e428190d3f7727f36d96c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 55830424a98348e382759c54afdeabcd. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:45.782052 28832 raft_consensus.cc:2932] T 55830424a98348e382759c54afdeabcd P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:45.785245 19219 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 55830424a98348e382759c54afdeabcd with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:10:45.786389 21530 raft_consensus.cc:2932] T 55830424a98348e382759c54afdeabcd P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:45.786631 19230 catalog_manager.cc:5484] T 55830424a98348e382759c54afdeabcd P 99a0be6ef2144b7fb247ba9d3f051d66 reported cstate change: config changed from index 6 to 7, VOTER 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133) added. New cstate: current_term: 1 leader_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:45.791576 28856 ts_tablet_manager.cc:902] T 11f64b59f83e458f8cf51fd5bcd36fee P 5bf81f3d986e428190d3f7727f36d96c: Initiating tablet copy from peer da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357)
I20240501 14:10:45.792444 28856 tablet_copy_client.cc:308] T 11f64b59f83e458f8cf51fd5bcd36fee P 5bf81f3d986e428190d3f7727f36d96c: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.132:35357
I20240501 14:10:45.793272 19789 tablet_copy_service.cc:140] P da3c4e948ff747678f18456c134d7ed4: Received BeginTabletCopySession request for tablet 11f64b59f83e458f8cf51fd5bcd36fee from peer 5bf81f3d986e428190d3f7727f36d96c ({username='slave'} at 127.7.39.133:51567)
I20240501 14:10:45.793370 19789 tablet_copy_service.cc:161] P da3c4e948ff747678f18456c134d7ed4: Beginning new tablet copy session on tablet 11f64b59f83e458f8cf51fd5bcd36fee from peer 5bf81f3d986e428190d3f7727f36d96c at {username='slave'} at 127.7.39.133:51567: session id = 5bf81f3d986e428190d3f7727f36d96c-11f64b59f83e458f8cf51fd5bcd36fee
I20240501 14:10:45.794312 19789 tablet_copy_source_session.cc:205] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:45.795209 28856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11f64b59f83e458f8cf51fd5bcd36fee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:45.818724 28856 tablet_copy_client.cc:783] T 11f64b59f83e458f8cf51fd5bcd36fee P 5bf81f3d986e428190d3f7727f36d96c: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:45.818909 28856 tablet_copy_client.cc:647] T 11f64b59f83e458f8cf51fd5bcd36fee P 5bf81f3d986e428190d3f7727f36d96c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:45.822499 28856 tablet_copy_client.cc:523] T 11f64b59f83e458f8cf51fd5bcd36fee P 5bf81f3d986e428190d3f7727f36d96c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:45.824049 28856 tablet_bootstrap.cc:492] T 11f64b59f83e458f8cf51fd5bcd36fee P 5bf81f3d986e428190d3f7727f36d96c: Bootstrap starting.
I20240501 14:10:45.848457 19767 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cf62635e9a994259b9e293e5eb425b55"
dest_uuid: "da3c4e948ff747678f18456c134d7ed4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50488
I20240501 14:10:45.848649 19767 raft_consensus.cc:595] T cf62635e9a994259b9e293e5eb425b55 P da3c4e948ff747678f18456c134d7ed4 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:10:45.858994 28856 tablet_bootstrap.cc:492] T 11f64b59f83e458f8cf51fd5bcd36fee P 5bf81f3d986e428190d3f7727f36d96c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:45.859356 28856 tablet_bootstrap.cc:492] T 11f64b59f83e458f8cf51fd5bcd36fee P 5bf81f3d986e428190d3f7727f36d96c: Bootstrap complete.
I20240501 14:10:45.859575 28856 ts_tablet_manager.cc:1367] T 11f64b59f83e458f8cf51fd5bcd36fee P 5bf81f3d986e428190d3f7727f36d96c: Time spent bootstrapping tablet: real 0.036s user 0.030s sys 0.000s
I20240501 14:10:45.860067 28856 raft_consensus.cc:348] T 11f64b59f83e458f8cf51fd5bcd36fee P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:45.860196 28856 raft_consensus.cc:729] T 11f64b59f83e458f8cf51fd5bcd36fee P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf81f3d986e428190d3f7727f36d96c, State: Initialized, Role: FOLLOWER
I20240501 14:10:45.860334 28856 consensus_queue.cc:260] T 11f64b59f83e458f8cf51fd5bcd36fee P 5bf81f3d986e428190d3f7727f36d96c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:45.860786 28856 ts_tablet_manager.cc:1398] T 11f64b59f83e458f8cf51fd5bcd36fee P 5bf81f3d986e428190d3f7727f36d96c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:45.861366 19789 tablet_copy_service.cc:342] P da3c4e948ff747678f18456c134d7ed4: Request end of tablet copy session 5bf81f3d986e428190d3f7727f36d96c-11f64b59f83e458f8cf51fd5bcd36fee received from {username='slave'} at 127.7.39.133:51567
I20240501 14:10:45.861470 19789 tablet_copy_service.cc:433] P da3c4e948ff747678f18456c134d7ed4: ending tablet copy session 5bf81f3d986e428190d3f7727f36d96c-11f64b59f83e458f8cf51fd5bcd36fee on tablet 11f64b59f83e458f8cf51fd5bcd36fee with peer 5bf81f3d986e428190d3f7727f36d96c
I20240501 14:10:45.863236 28934 ts_tablet_manager.cc:902] T 1d408b075d784953a57554bcdf19158a P 99a0be6ef2144b7fb247ba9d3f051d66: Initiating tablet copy from peer da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357)
I20240501 14:10:45.863781 28934 tablet_copy_client.cc:308] T 1d408b075d784953a57554bcdf19158a P 99a0be6ef2144b7fb247ba9d3f051d66: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.132:35357
I20240501 14:10:45.864207 19789 tablet_copy_service.cc:140] P da3c4e948ff747678f18456c134d7ed4: Received BeginTabletCopySession request for tablet 1d408b075d784953a57554bcdf19158a from peer 99a0be6ef2144b7fb247ba9d3f051d66 ({username='slave'} at 127.7.39.130:52281)
I20240501 14:10:45.864313 19789 tablet_copy_service.cc:161] P da3c4e948ff747678f18456c134d7ed4: Beginning new tablet copy session on tablet 1d408b075d784953a57554bcdf19158a from peer 99a0be6ef2144b7fb247ba9d3f051d66 at {username='slave'} at 127.7.39.130:52281: session id = 99a0be6ef2144b7fb247ba9d3f051d66-1d408b075d784953a57554bcdf19158a
I20240501 14:10:45.865325 19789 tablet_copy_source_session.cc:205] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:45.866259 28934 ts_tablet_manager.cc:1875] T 1d408b075d784953a57554bcdf19158a P 99a0be6ef2144b7fb247ba9d3f051d66: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:10:45.870370 28934 ts_tablet_manager.cc:1888] T 1d408b075d784953a57554bcdf19158a P 99a0be6ef2144b7fb247ba9d3f051d66: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 14:10:45.870558 28934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d408b075d784953a57554bcdf19158a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:45.872205 28934 tablet_copy_client.cc:783] T 1d408b075d784953a57554bcdf19158a P 99a0be6ef2144b7fb247ba9d3f051d66: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:45.872354 28934 tablet_copy_client.cc:647] T 1d408b075d784953a57554bcdf19158a P 99a0be6ef2144b7fb247ba9d3f051d66: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:45.873530 28934 tablet_copy_client.cc:523] T 1d408b075d784953a57554bcdf19158a P 99a0be6ef2144b7fb247ba9d3f051d66: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:45.874727 28934 tablet_bootstrap.cc:492] T 1d408b075d784953a57554bcdf19158a P 99a0be6ef2144b7fb247ba9d3f051d66: Bootstrap starting.
I20240501 14:10:45.878691 21398 consensus_queue.cc:237] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:45.879927 21136 raft_consensus.cc:1261] T f69020bb6c244848a6b0a50fd2b4f3eb P 5bf81f3d986e428190d3f7727f36d96c [term 2 FOLLOWER]: Refusing update from remote peer 43ce4a4c954f4a5f80954b38933c189d: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:10:45.880569 28214 consensus_queue.cc:1035] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:10:45.884248 28214 raft_consensus.cc:2932] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:45.885998 21136 raft_consensus.cc:2932] T f69020bb6c244848a6b0a50fd2b4f3eb P 5bf81f3d986e428190d3f7727f36d96c [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:45.888701 19230 catalog_manager.cc:5484] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d reported cstate change: config changed from index 7 to 9, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) evicted. New cstate: current_term: 2 leader_uuid: "43ce4a4c954f4a5f80954b38933c189d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:45.892048 19356 tablet_service.cc:1514] Processing DeleteTablet for tablet f69020bb6c244848a6b0a50fd2b4f3eb with delete_type TABLET_DATA_TOMBSTONED (TS fb1e6bc2d07b4ce3b622a66f14c93bdc not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:46838
I20240501 14:10:45.892432 28970 tablet_replica.cc:330] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc: stopping tablet replica
I20240501 14:10:45.892608 28970 raft_consensus.cc:2229] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:45.892968 28970 raft_consensus.cc:2258] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:45.893728 28970 ts_tablet_manager.cc:1875] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:45.904886 28856 ts_tablet_manager.cc:902] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c: Initiating tablet copy from peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911)
I20240501 14:10:45.905413 28856 tablet_copy_client.cc:308] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.134:41911
I20240501 14:10:45.907130 28970 ts_tablet_manager.cc:1888] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.8
I20240501 14:10:45.907361 28970 log.cc:1196] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal/wals/f69020bb6c244848a6b0a50fd2b4f3eb
I20240501 14:10:45.909456 19218 catalog_manager.cc:4830] TS fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847): tablet f69020bb6c244848a6b0a50fd2b4f3eb (table rebalance_test_table_4 [id=151dfad0ef40470ab9c5203c42400b60]) successfully deleted
I20240501 14:10:45.911310 21288 tablet_copy_service.cc:140] P e6246b183293412494919f1e7efc9448: Received BeginTabletCopySession request for tablet 814ba2247a314a4797aa0f9f624aaeb6 from peer 5bf81f3d986e428190d3f7727f36d96c ({username='slave'} at 127.7.39.133:57713)
I20240501 14:10:45.911456 21288 tablet_copy_service.cc:161] P e6246b183293412494919f1e7efc9448: Beginning new tablet copy session on tablet 814ba2247a314a4797aa0f9f624aaeb6 from peer 5bf81f3d986e428190d3f7727f36d96c at {username='slave'} at 127.7.39.133:57713: session id = 5bf81f3d986e428190d3f7727f36d96c-814ba2247a314a4797aa0f9f624aaeb6
I20240501 14:10:45.912860 21288 tablet_copy_source_session.cc:205] T 814ba2247a314a4797aa0f9f624aaeb6 P e6246b183293412494919f1e7efc9448: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:45.914347 28856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 814ba2247a314a4797aa0f9f624aaeb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:45.915375 21398 consensus_queue.cc:237] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:45.916620 28856 tablet_copy_client.cc:783] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:45.916795 28856 tablet_copy_client.cc:647] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:45.917342 21136 raft_consensus.cc:1261] T f69020bb6c244848a6b0a50fd2b4f3eb P 5bf81f3d986e428190d3f7727f36d96c [term 2 FOLLOWER]: Refusing update from remote peer 43ce4a4c954f4a5f80954b38933c189d: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 14:10:45.918116 28214 consensus_queue.cc:1035] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:10:45.921651 28856 tablet_copy_client.cc:523] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:45.922044 28214 raft_consensus.cc:2932] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:45.923025 28934 tablet_bootstrap.cc:492] T 1d408b075d784953a57554bcdf19158a P 99a0be6ef2144b7fb247ba9d3f051d66: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:45.923111 28856 tablet_bootstrap.cc:492] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c: Bootstrap starting.
I20240501 14:10:45.923372 28934 tablet_bootstrap.cc:492] T 1d408b075d784953a57554bcdf19158a P 99a0be6ef2144b7fb247ba9d3f051d66: Bootstrap complete.
I20240501 14:10:45.923547 28934 ts_tablet_manager.cc:1367] T 1d408b075d784953a57554bcdf19158a P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent bootstrapping tablet: real 0.049s user 0.027s sys 0.001s
I20240501 14:10:45.923684 21136 raft_consensus.cc:2932] T f69020bb6c244848a6b0a50fd2b4f3eb P 5bf81f3d986e428190d3f7727f36d96c [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:45.924005 28934 raft_consensus.cc:348] T 1d408b075d784953a57554bcdf19158a P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:45.924134 28934 raft_consensus.cc:729] T 1d408b075d784953a57554bcdf19158a P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a0be6ef2144b7fb247ba9d3f051d66, State: Initialized, Role: FOLLOWER
I20240501 14:10:45.924304 28934 consensus_queue.cc:260] T 1d408b075d784953a57554bcdf19158a P 99a0be6ef2144b7fb247ba9d3f051d66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:10:45.924697 28934 ts_tablet_manager.cc:1398] T 1d408b075d784953a57554bcdf19158a P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:10:45.926847 19789 tablet_copy_service.cc:342] P da3c4e948ff747678f18456c134d7ed4: Request end of tablet copy session 99a0be6ef2144b7fb247ba9d3f051d66-1d408b075d784953a57554bcdf19158a received from {username='slave'} at 127.7.39.130:52281
I20240501 14:10:45.926952 19789 tablet_copy_service.cc:433] P da3c4e948ff747678f18456c134d7ed4: ending tablet copy session 99a0be6ef2144b7fb247ba9d3f051d66-1d408b075d784953a57554bcdf19158a on tablet 1d408b075d784953a57554bcdf19158a with peer 99a0be6ef2144b7fb247ba9d3f051d66
I20240501 14:10:45.927726 19218 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet f69020bb6c244848a6b0a50fd2b4f3eb with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:10:45.928062 19227 catalog_manager.cc:5484] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d reported cstate change: config changed from index 9 to 10, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New cstate: current_term: 2 leader_uuid: "43ce4a4c954f4a5f80954b38933c189d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
W20240501 14:10:45.942731 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f69020bb6c244848a6b0a50fd2b4f3eb. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:45.973690 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:45.977054 19767 raft_consensus.cc:1203] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:10:45.984488 21492 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:45.990824 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:45.999451 28856 tablet_bootstrap.cc:492] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:45.999775 28856 tablet_bootstrap.cc:492] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c: Bootstrap complete.
I20240501 14:10:45.999976 28856 ts_tablet_manager.cc:1367] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c: Time spent bootstrapping tablet: real 0.077s user 0.029s sys 0.000s
I20240501 14:10:46.000474 28856 raft_consensus.cc:348] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:46.000597 28856 raft_consensus.cc:729] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf81f3d986e428190d3f7727f36d96c, State: Initialized, Role: FOLLOWER
I20240501 14:10:46.001915 19617 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:10:46.003276 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:46.004315 28856 consensus_queue.cc:260] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:46.006441 28856 ts_tablet_manager.cc:1398] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:10:46.007390 21288 tablet_copy_service.cc:342] P e6246b183293412494919f1e7efc9448: Request end of tablet copy session 5bf81f3d986e428190d3f7727f36d96c-814ba2247a314a4797aa0f9f624aaeb6 received from {username='slave'} at 127.7.39.133:57713
I20240501 14:10:46.007488 21288 tablet_copy_service.cc:433] P e6246b183293412494919f1e7efc9448: ending tablet copy session 5bf81f3d986e428190d3f7727f36d96c-814ba2247a314a4797aa0f9f624aaeb6 on tablet 814ba2247a314a4797aa0f9f624aaeb6 with peer 5bf81f3d986e428190d3f7727f36d96c
I20240501 14:10:46.021936 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:46.022989 21247 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:46.029228 19356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:46.050238 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:46.073432 21492 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:46.083765 19356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:46.089008 21247 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:46.089776 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:46.092486 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:46.120676 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:46.124082 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:46.129885 19748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:46.132975 21492 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:46.133864 19617 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:10:46.161077 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:46.174110 21247 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:46.192924 19356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:46.195425 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:46.199923 19617 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:10:46.233851 28856 ts_tablet_manager.cc:902] T 55830424a98348e382759c54afdeabcd P 5bf81f3d986e428190d3f7727f36d96c: Initiating tablet copy from peer 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829)
I20240501 14:10:46.234443 28856 tablet_copy_client.cc:308] T 55830424a98348e382759c54afdeabcd P 5bf81f3d986e428190d3f7727f36d96c: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.130:41829
I20240501 14:10:46.240710 19527 tablet_copy_service.cc:140] P 99a0be6ef2144b7fb247ba9d3f051d66: Received BeginTabletCopySession request for tablet 55830424a98348e382759c54afdeabcd from peer 5bf81f3d986e428190d3f7727f36d96c ({username='slave'} at 127.7.39.133:45323)
I20240501 14:10:46.240856 19527 tablet_copy_service.cc:161] P 99a0be6ef2144b7fb247ba9d3f051d66: Beginning new tablet copy session on tablet 55830424a98348e382759c54afdeabcd from peer 5bf81f3d986e428190d3f7727f36d96c at {username='slave'} at 127.7.39.133:45323: session id = 5bf81f3d986e428190d3f7727f36d96c-55830424a98348e382759c54afdeabcd
I20240501 14:10:46.241870 19527 tablet_copy_source_session.cc:205] T 55830424a98348e382759c54afdeabcd P 99a0be6ef2144b7fb247ba9d3f051d66: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:46.242961 29058 raft_consensus.cc:979] T cf62635e9a994259b9e293e5eb425b55 P da3c4e948ff747678f18456c134d7ed4: : Instructing follower ec8cde17a00645e7b38d5d29f7425367 to start an election
I20240501 14:10:46.243044 29058 raft_consensus.cc:1067] T cf62635e9a994259b9e293e5eb425b55 P da3c4e948ff747678f18456c134d7ed4 [term 1 LEADER]: Signalling peer ec8cde17a00645e7b38d5d29f7425367 to start an election
I20240501 14:10:46.243654 19637 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cf62635e9a994259b9e293e5eb425b55"
dest_uuid: "ec8cde17a00645e7b38d5d29f7425367"
from {username='slave'} at 127.7.39.132:44167
I20240501 14:10:46.243803 19637 raft_consensus.cc:482] T cf62635e9a994259b9e293e5eb425b55 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:10:46.243867 19637 raft_consensus.cc:3037] T cf62635e9a994259b9e293e5eb425b55 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:46.244988 28856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55830424a98348e382759c54afdeabcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:46.245146 19637 raft_consensus.cc:504] T cf62635e9a994259b9e293e5eb425b55 P ec8cde17a00645e7b38d5d29f7425367 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:10:46.245674 19637 leader_election.cc:288] T cf62635e9a994259b9e293e5eb425b55 P ec8cde17a00645e7b38d5d29f7425367 [CANDIDATE]: Term 2 election: Requested vote from peers da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357), 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136:34895)
I20240501 14:10:46.246659 21530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf62635e9a994259b9e293e5eb425b55" candidate_uuid: "ec8cde17a00645e7b38d5d29f7425367" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "778c78105092428bbcde75ab2cdb6a05"
I20240501 14:10:46.246821 19769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf62635e9a994259b9e293e5eb425b55" candidate_uuid: "ec8cde17a00645e7b38d5d29f7425367" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "da3c4e948ff747678f18456c134d7ed4"
I20240501 14:10:46.246848 21530 raft_consensus.cc:3037] T cf62635e9a994259b9e293e5eb425b55 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:46.246951 19769 raft_consensus.cc:3032] T cf62635e9a994259b9e293e5eb425b55 P da3c4e948ff747678f18456c134d7ed4 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:10:46.247001 19769 raft_consensus.cc:729] T cf62635e9a994259b9e293e5eb425b55 P da3c4e948ff747678f18456c134d7ed4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: da3c4e948ff747678f18456c134d7ed4, State: Running, Role: LEADER
I20240501 14:10:46.247221 19769 consensus_queue.cc:260] T cf62635e9a994259b9e293e5eb425b55 P da3c4e948ff747678f18456c134d7ed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:10:46.247426 19769 raft_consensus.cc:3037] T cf62635e9a994259b9e293e5eb425b55 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:46.248044 21530 raft_consensus.cc:2454] T cf62635e9a994259b9e293e5eb425b55 P 778c78105092428bbcde75ab2cdb6a05 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec8cde17a00645e7b38d5d29f7425367 in term 2.
I20240501 14:10:46.248395 28856 tablet_copy_client.cc:783] T 55830424a98348e382759c54afdeabcd P 5bf81f3d986e428190d3f7727f36d96c: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:46.248584 19769 raft_consensus.cc:2454] T cf62635e9a994259b9e293e5eb425b55 P da3c4e948ff747678f18456c134d7ed4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec8cde17a00645e7b38d5d29f7425367 in term 2.
I20240501 14:10:46.248690 28856 tablet_copy_client.cc:647] T 55830424a98348e382759c54afdeabcd P 5bf81f3d986e428190d3f7727f36d96c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:46.249078 19573 leader_election.cc:302] T cf62635e9a994259b9e293e5eb425b55 P ec8cde17a00645e7b38d5d29f7425367 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da3c4e948ff747678f18456c134d7ed4, ec8cde17a00645e7b38d5d29f7425367; no voters:
I20240501 14:10:46.249281 29109 raft_consensus.cc:2781] T cf62635e9a994259b9e293e5eb425b55 P ec8cde17a00645e7b38d5d29f7425367 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:10:46.249384 29109 raft_consensus.cc:686] T cf62635e9a994259b9e293e5eb425b55 P ec8cde17a00645e7b38d5d29f7425367 [term 2 LEADER]: Becoming Leader. State: Replica: ec8cde17a00645e7b38d5d29f7425367, State: Running, Role: LEADER
I20240501 14:10:46.249540 29109 consensus_queue.cc:237] T cf62635e9a994259b9e293e5eb425b55 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:10:46.250046 28856 tablet_copy_client.cc:523] T 55830424a98348e382759c54afdeabcd P 5bf81f3d986e428190d3f7727f36d96c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:46.252036 19227 catalog_manager.cc:5484] T cf62635e9a994259b9e293e5eb425b55 P ec8cde17a00645e7b38d5d29f7425367 reported cstate change: term changed from 1 to 2, leader changed from da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) to ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131). New cstate: current_term: 2 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:46.253002 28856 tablet_bootstrap.cc:492] T 55830424a98348e382759c54afdeabcd P 5bf81f3d986e428190d3f7727f36d96c: Bootstrap starting.
I20240501 14:10:46.254501 19506 raft_consensus.cc:1203] T 1d408b075d784953a57554bcdf19158a P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:10:46.287520 28856 tablet_bootstrap.cc:492] T 55830424a98348e382759c54afdeabcd P 5bf81f3d986e428190d3f7727f36d96c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:46.287834 28856 tablet_bootstrap.cc:492] T 55830424a98348e382759c54afdeabcd P 5bf81f3d986e428190d3f7727f36d96c: Bootstrap complete.
I20240501 14:10:46.288079 28856 ts_tablet_manager.cc:1367] T 55830424a98348e382759c54afdeabcd P 5bf81f3d986e428190d3f7727f36d96c: Time spent bootstrapping tablet: real 0.037s user 0.031s sys 0.000s
I20240501 14:10:46.288580 28856 raft_consensus.cc:348] T 55830424a98348e382759c54afdeabcd P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:46.288712 28856 raft_consensus.cc:729] T 55830424a98348e382759c54afdeabcd P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf81f3d986e428190d3f7727f36d96c, State: Initialized, Role: FOLLOWER
I20240501 14:10:46.288843 28856 consensus_queue.cc:260] T 55830424a98348e382759c54afdeabcd P 5bf81f3d986e428190d3f7727f36d96c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:46.290030 28856 ts_tablet_manager.cc:1398] T 55830424a98348e382759c54afdeabcd P 5bf81f3d986e428190d3f7727f36d96c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:10:46.293828 19527 tablet_copy_service.cc:342] P 99a0be6ef2144b7fb247ba9d3f051d66: Request end of tablet copy session 5bf81f3d986e428190d3f7727f36d96c-55830424a98348e382759c54afdeabcd received from {username='slave'} at 127.7.39.133:45323
I20240501 14:10:46.293922 19527 tablet_copy_service.cc:433] P 99a0be6ef2144b7fb247ba9d3f051d66: ending tablet copy session 5bf81f3d986e428190d3f7727f36d96c-55830424a98348e382759c54afdeabcd on tablet 55830424a98348e382759c54afdeabcd with peer 5bf81f3d986e428190d3f7727f36d96c
I20240501 14:10:46.325492 19506 consensus_queue.cc:237] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:10:46.326963 21530 raft_consensus.cc:1261] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Refusing update from remote peer 99a0be6ef2144b7fb247ba9d3f051d66: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:10:46.327515 27152 consensus_queue.cc:1035] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:10:46.345774 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:46.348321 27152 raft_consensus.cc:2932] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:46.352813 19233 catalog_manager.cc:5484] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 reported cstate change: config changed from index 5 to 6, VOTER ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131) evicted. New cstate: current_term: 1 leader_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:46.352978 21530 raft_consensus.cc:2932] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:46.355471 19617 tablet_service.cc:1514] Processing DeleteTablet for tablet 272eee3450cf4f48a4e1de2dcb28c0f7 with delete_type TABLET_DATA_TOMBSTONED (TS ec8cde17a00645e7b38d5d29f7425367 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47238
I20240501 14:10:46.356609 29285 tablet_replica.cc:330] T 272eee3450cf4f48a4e1de2dcb28c0f7 P ec8cde17a00645e7b38d5d29f7425367: stopping tablet replica
I20240501 14:10:46.357039 29285 raft_consensus.cc:2229] T 272eee3450cf4f48a4e1de2dcb28c0f7 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:46.357072 19506 consensus_queue.cc:237] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:46.357276 29285 raft_consensus.cc:2258] T 272eee3450cf4f48a4e1de2dcb28c0f7 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:46.358592 29285 ts_tablet_manager.cc:1875] T 272eee3450cf4f48a4e1de2dcb28c0f7 P ec8cde17a00645e7b38d5d29f7425367: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:46.359328 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:46.361900 29285 ts_tablet_manager.cc:1888] T 272eee3450cf4f48a4e1de2dcb28c0f7 P ec8cde17a00645e7b38d5d29f7425367: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:10:46.362123 29285 log.cc:1196] T 272eee3450cf4f48a4e1de2dcb28c0f7 P ec8cde17a00645e7b38d5d29f7425367: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal/wals/272eee3450cf4f48a4e1de2dcb28c0f7
I20240501 14:10:46.362818 19219 catalog_manager.cc:4830] TS ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201): tablet 272eee3450cf4f48a4e1de2dcb28c0f7 (table rebalance_test_table_7 [id=ed1a7ed87fa9480ba18a052130ecdc05]) successfully deleted
I20240501 14:10:46.367823 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:46.369432 21530 raft_consensus.cc:1261] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Refusing update from remote peer 99a0be6ef2144b7fb247ba9d3f051d66: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:10:46.370237 27152 consensus_queue.cc:1035] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:10:46.372123 27152 raft_consensus.cc:2932] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:46.374313 19219 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 272eee3450cf4f48a4e1de2dcb28c0f7 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:10:46.374768 21530 raft_consensus.cc:2932] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:46.378113 19229 catalog_manager.cc:5484] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 reported cstate change: config changed from index 6 to 7, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New cstate: current_term: 1 leader_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:46.379905 21492 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:46.389490 21492 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:46.391086 21247 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:46.398228 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:46.399597 21136 raft_consensus.cc:1203] T 11f64b59f83e458f8cf51fd5bcd36fee P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:10:46.409760 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:46.410169 19356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:46.424687 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:46.425740 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:46.433800 19356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:46.444545 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:46.468232 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:46.477190 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:46.478829 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:46.484508 21492 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:46.485885 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:46.495859 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:46.507814 21136 raft_consensus.cc:1203] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20240501 14:10:46.517104 29315 ts_tablet_manager.cc:902] T f69020bb6c244848a6b0a50fd2b4f3eb P e6246b183293412494919f1e7efc9448: Initiating tablet copy from peer 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135:38751)
I20240501 14:10:46.519008 29315 tablet_copy_client.cc:308] T f69020bb6c244848a6b0a50fd2b4f3eb P e6246b183293412494919f1e7efc9448: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.135:38751
I20240501 14:10:46.519259 19356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:46.519737 21419 tablet_copy_service.cc:140] P 43ce4a4c954f4a5f80954b38933c189d: Received BeginTabletCopySession request for tablet f69020bb6c244848a6b0a50fd2b4f3eb from peer e6246b183293412494919f1e7efc9448 ({username='slave'} at 127.7.39.134:38199)
I20240501 14:10:46.519892 21419 tablet_copy_service.cc:161] P 43ce4a4c954f4a5f80954b38933c189d: Beginning new tablet copy session on tablet f69020bb6c244848a6b0a50fd2b4f3eb from peer e6246b183293412494919f1e7efc9448 at {username='slave'} at 127.7.39.134:38199: session id = e6246b183293412494919f1e7efc9448-f69020bb6c244848a6b0a50fd2b4f3eb
I20240501 14:10:46.520895 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:46.521323 21419 tablet_copy_source_session.cc:205] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:46.522567 29315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f69020bb6c244848a6b0a50fd2b4f3eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:46.524273 21247 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:46.529520 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 272eee3450cf4f48a4e1de2dcb28c0f7. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:46.532270 21247 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:46.557637 29315 tablet_copy_client.cc:783] T f69020bb6c244848a6b0a50fd2b4f3eb P e6246b183293412494919f1e7efc9448: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:46.557919 29315 tablet_copy_client.cc:647] T f69020bb6c244848a6b0a50fd2b4f3eb P e6246b183293412494919f1e7efc9448: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:46.563200 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:46.563560 29315 tablet_copy_client.cc:523] T f69020bb6c244848a6b0a50fd2b4f3eb P e6246b183293412494919f1e7efc9448: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:46.565316 29315 tablet_bootstrap.cc:492] T f69020bb6c244848a6b0a50fd2b4f3eb P e6246b183293412494919f1e7efc9448: Bootstrap starting.
I20240501 14:10:46.583221 19769 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ec2a46f8cfb3406082970e5fdb59b101"
dest_uuid: "da3c4e948ff747678f18456c134d7ed4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50644
I20240501 14:10:46.583377 19769 raft_consensus.cc:595] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:10:46.611774 29315 tablet_bootstrap.cc:492] T f69020bb6c244848a6b0a50fd2b4f3eb P e6246b183293412494919f1e7efc9448: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:46.612119 29315 tablet_bootstrap.cc:492] T f69020bb6c244848a6b0a50fd2b4f3eb P e6246b183293412494919f1e7efc9448: Bootstrap complete.
I20240501 14:10:46.612306 29315 ts_tablet_manager.cc:1367] T f69020bb6c244848a6b0a50fd2b4f3eb P e6246b183293412494919f1e7efc9448: Time spent bootstrapping tablet: real 0.047s user 0.029s sys 0.003s
I20240501 14:10:46.612721 29315 raft_consensus.cc:348] T f69020bb6c244848a6b0a50fd2b4f3eb P e6246b183293412494919f1e7efc9448 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:46.612855 29315 raft_consensus.cc:729] T f69020bb6c244848a6b0a50fd2b4f3eb P e6246b183293412494919f1e7efc9448 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6246b183293412494919f1e7efc9448, State: Initialized, Role: FOLLOWER
I20240501 14:10:46.612973 29315 consensus_queue.cc:260] T f69020bb6c244848a6b0a50fd2b4f3eb P e6246b183293412494919f1e7efc9448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:46.613216 29315 ts_tablet_manager.cc:1398] T f69020bb6c244848a6b0a50fd2b4f3eb P e6246b183293412494919f1e7efc9448: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:10:46.613790 21419 tablet_copy_service.cc:342] P 43ce4a4c954f4a5f80954b38933c189d: Request end of tablet copy session e6246b183293412494919f1e7efc9448-f69020bb6c244848a6b0a50fd2b4f3eb received from {username='slave'} at 127.7.39.134:38199
I20240501 14:10:46.613884 21419 tablet_copy_service.cc:433] P 43ce4a4c954f4a5f80954b38933c189d: ending tablet copy session e6246b183293412494919f1e7efc9448-f69020bb6c244848a6b0a50fd2b4f3eb on tablet f69020bb6c244848a6b0a50fd2b4f3eb with peer e6246b183293412494919f1e7efc9448
I20240501 14:10:46.625666 19375 consensus_queue.cc:237] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:10:46.626905 21530 raft_consensus.cc:1261] T 410a7dd0dd734684a82875912966657d P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:10:46.627321 29316 consensus_queue.cc:1035] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:10:46.628960 19375 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "98af8b06b9324e3fa7cdb898b7b9fab9"
dest_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34588
I20240501 14:10:46.629071 19375 raft_consensus.cc:595] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Received request to transfer leadership
I20240501 14:10:46.630970 28229 raft_consensus.cc:2932] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:46.635834 21530 raft_consensus.cc:2932] T 410a7dd0dd734684a82875912966657d P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:46.636201 19229 catalog_manager.cc:5484] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc reported cstate change: config changed from index 7 to 8, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New cstate: current_term: 1 leader_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:46.639684 19748 tablet_service.cc:1514] Processing DeleteTablet for tablet 410a7dd0dd734684a82875912966657d with delete_type TABLET_DATA_TOMBSTONED (TS da3c4e948ff747678f18456c134d7ed4 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:36530
I20240501 14:10:46.640936 19375 consensus_queue.cc:237] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:46.641850 29393 tablet_replica.cc:330] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4: stopping tablet replica
I20240501 14:10:46.642000 29393 raft_consensus.cc:2229] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:46.642113 29393 raft_consensus.cc:2258] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:46.642658 29393 ts_tablet_manager.cc:1875] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:46.644558 28670 raft_consensus.cc:979] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4: : Instructing follower 778c78105092428bbcde75ab2cdb6a05 to start an election
I20240501 14:10:46.644645 28670 raft_consensus.cc:1067] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4 [term 1 LEADER]: Signalling peer 778c78105092428bbcde75ab2cdb6a05 to start an election
I20240501 14:10:46.645218 21530 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ec2a46f8cfb3406082970e5fdb59b101"
dest_uuid: "778c78105092428bbcde75ab2cdb6a05"
from {username='slave'} at 127.7.39.132:41311
I20240501 14:10:46.645409 21530 raft_consensus.cc:482] T ec2a46f8cfb3406082970e5fdb59b101 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:10:46.645483 21530 raft_consensus.cc:3037] T ec2a46f8cfb3406082970e5fdb59b101 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:46.645467 29393 ts_tablet_manager.cc:1888] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 14:10:46.645663 29393 log.cc:1196] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal/wals/410a7dd0dd734684a82875912966657d
I20240501 14:10:46.646562 21530 raft_consensus.cc:504] T ec2a46f8cfb3406082970e5fdb59b101 P 778c78105092428bbcde75ab2cdb6a05 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:10:46.647837 21530 leader_election.cc:288] T ec2a46f8cfb3406082970e5fdb59b101 P 778c78105092428bbcde75ab2cdb6a05 [CANDIDATE]: Term 2 election: Requested vote from peers da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357), fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847), 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133:46883)
I20240501 14:10:46.648661 21530 raft_consensus.cc:1261] T 410a7dd0dd734684a82875912966657d P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:10:46.649055 29391 consensus_queue.cc:1035] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:10:46.649988 19220 catalog_manager.cc:4830] TS da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357): tablet 410a7dd0dd734684a82875912966657d (table rebalance_test_table_2 [id=6f6b0a378a484e5889d733ff982d9eb7]) successfully deleted
I20240501 14:10:46.650305 19769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec2a46f8cfb3406082970e5fdb59b101" candidate_uuid: "778c78105092428bbcde75ab2cdb6a05" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "da3c4e948ff747678f18456c134d7ed4"
I20240501 14:10:46.650426 19769 raft_consensus.cc:3032] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:10:46.650471 19769 raft_consensus.cc:729] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: da3c4e948ff747678f18456c134d7ed4, State: Running, Role: LEADER
I20240501 14:10:46.650600 19769 consensus_queue.cc:260] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:10:46.650749 19769 raft_consensus.cc:3037] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:46.651712 19769 raft_consensus.cc:2454] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 778c78105092428bbcde75ab2cdb6a05 in term 2.
I20240501 14:10:46.651659 29391 raft_consensus.cc:2932] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:46.653367 21136 raft_consensus.cc:1203] T 55830424a98348e382759c54afdeabcd P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:10:46.654392 21530 raft_consensus.cc:2932] T 410a7dd0dd734684a82875912966657d P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:46.655030 19229 catalog_manager.cc:5484] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc reported cstate change: config changed from index 8 to 9, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) added. New cstate: current_term: 1 leader_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
W20240501 14:10:46.655797 19310 consensus_peers.cc:485] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 410a7dd0dd734684a82875912966657d. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:46.656472 19218 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 410a7dd0dd734684a82875912966657d with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:10:46.661684 19375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec2a46f8cfb3406082970e5fdb59b101" candidate_uuid: "778c78105092428bbcde75ab2cdb6a05" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc"
I20240501 14:10:46.661819 19375 raft_consensus.cc:3037] T ec2a46f8cfb3406082970e5fdb59b101 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:46.662956 19375 raft_consensus.cc:2454] T ec2a46f8cfb3406082970e5fdb59b101 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 778c78105092428bbcde75ab2cdb6a05 in term 2.
I20240501 14:10:46.664233 21463 leader_election.cc:302] T ec2a46f8cfb3406082970e5fdb59b101 P 778c78105092428bbcde75ab2cdb6a05 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 778c78105092428bbcde75ab2cdb6a05, da3c4e948ff747678f18456c134d7ed4, fb1e6bc2d07b4ce3b622a66f14c93bdc; no voters:
I20240501 14:10:46.664533 29284 raft_consensus.cc:2781] T ec2a46f8cfb3406082970e5fdb59b101 P 778c78105092428bbcde75ab2cdb6a05 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:10:46.664726 29284 raft_consensus.cc:686] T ec2a46f8cfb3406082970e5fdb59b101 P 778c78105092428bbcde75ab2cdb6a05 [term 2 LEADER]: Becoming Leader. State: Replica: 778c78105092428bbcde75ab2cdb6a05, State: Running, Role: LEADER
I20240501 14:10:46.665091 29284 consensus_queue.cc:237] T ec2a46f8cfb3406082970e5fdb59b101 P 778c78105092428bbcde75ab2cdb6a05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:10:46.672860 21136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec2a46f8cfb3406082970e5fdb59b101" candidate_uuid: "778c78105092428bbcde75ab2cdb6a05" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "5bf81f3d986e428190d3f7727f36d96c"
I20240501 14:10:46.673012 21136 raft_consensus.cc:3037] T ec2a46f8cfb3406082970e5fdb59b101 P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:46.673630 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:46.674226 21136 raft_consensus.cc:2454] T ec2a46f8cfb3406082970e5fdb59b101 P 5bf81f3d986e428190d3f7727f36d96c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 778c78105092428bbcde75ab2cdb6a05 in term 2.
I20240501 14:10:46.675560 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:46.676330 19748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:46.677634 19225 catalog_manager.cc:5484] T ec2a46f8cfb3406082970e5fdb59b101 P 778c78105092428bbcde75ab2cdb6a05 reported cstate change: term changed from 1 to 2, leader changed from da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) to 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136). New cstate: current_term: 2 leader_uuid: "778c78105092428bbcde75ab2cdb6a05" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:46.691125 19356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:46.692044 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:46.692627 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:46.720265 19769 raft_consensus.cc:1261] T cf62635e9a994259b9e293e5eb425b55 P da3c4e948ff747678f18456c134d7ed4 [term 2 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:10:46.721243 21247 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:46.721541 28151 consensus_queue.cc:1035] T cf62635e9a994259b9e293e5eb425b55 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:10:46.746975 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:46.751883 21530 raft_consensus.cc:1261] T cf62635e9a994259b9e293e5eb425b55 P 778c78105092428bbcde75ab2cdb6a05 [term 2 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:10:46.752883 29109 consensus_queue.cc:1035] T cf62635e9a994259b9e293e5eb425b55 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:10:46.787569 29315 ts_tablet_manager.cc:902] T 272eee3450cf4f48a4e1de2dcb28c0f7 P e6246b183293412494919f1e7efc9448: Initiating tablet copy from peer 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829)
I20240501 14:10:46.788000 29315 tablet_copy_client.cc:308] T 272eee3450cf4f48a4e1de2dcb28c0f7 P e6246b183293412494919f1e7efc9448: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.130:41829
I20240501 14:10:46.794072 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:46.814875 19748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:46.819571 21247 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:46.830541 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:46.831416 19356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:46.846655 19527 tablet_copy_service.cc:140] P 99a0be6ef2144b7fb247ba9d3f051d66: Received BeginTabletCopySession request for tablet 272eee3450cf4f48a4e1de2dcb28c0f7 from peer e6246b183293412494919f1e7efc9448 ({username='slave'} at 127.7.39.134:39861)
I20240501 14:10:46.846789 19527 tablet_copy_service.cc:161] P 99a0be6ef2144b7fb247ba9d3f051d66: Beginning new tablet copy session on tablet 272eee3450cf4f48a4e1de2dcb28c0f7 from peer e6246b183293412494919f1e7efc9448 at {username='slave'} at 127.7.39.134:39861: session id = e6246b183293412494919f1e7efc9448-272eee3450cf4f48a4e1de2dcb28c0f7
I20240501 14:10:46.847210 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:46.847828 19527 tablet_copy_source_session.cc:205] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:46.848960 29315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 272eee3450cf4f48a4e1de2dcb28c0f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:46.851248 29315 tablet_copy_client.cc:783] T 272eee3450cf4f48a4e1de2dcb28c0f7 P e6246b183293412494919f1e7efc9448: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:46.851428 29315 tablet_copy_client.cc:647] T 272eee3450cf4f48a4e1de2dcb28c0f7 P e6246b183293412494919f1e7efc9448: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:46.855569 29315 tablet_copy_client.cc:523] T 272eee3450cf4f48a4e1de2dcb28c0f7 P e6246b183293412494919f1e7efc9448: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:46.857120 29315 tablet_bootstrap.cc:492] T 272eee3450cf4f48a4e1de2dcb28c0f7 P e6246b183293412494919f1e7efc9448: Bootstrap starting.
I20240501 14:10:46.860320 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:46.863924 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:46.926292 21117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:10:46.930433 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:46.940755 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:46.946812 19748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:46.949268 29315 tablet_bootstrap.cc:492] T 272eee3450cf4f48a4e1de2dcb28c0f7 P e6246b183293412494919f1e7efc9448: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:46.949712 29315 tablet_bootstrap.cc:492] T 272eee3450cf4f48a4e1de2dcb28c0f7 P e6246b183293412494919f1e7efc9448: Bootstrap complete.
I20240501 14:10:46.949932 29315 ts_tablet_manager.cc:1367] T 272eee3450cf4f48a4e1de2dcb28c0f7 P e6246b183293412494919f1e7efc9448: Time spent bootstrapping tablet: real 0.093s user 0.030s sys 0.000s
I20240501 14:10:46.950480 29315 raft_consensus.cc:348] T 272eee3450cf4f48a4e1de2dcb28c0f7 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:46.950767 29315 raft_consensus.cc:729] T 272eee3450cf4f48a4e1de2dcb28c0f7 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6246b183293412494919f1e7efc9448, State: Initialized, Role: FOLLOWER
I20240501 14:10:46.950978 29315 consensus_queue.cc:260] T 272eee3450cf4f48a4e1de2dcb28c0f7 P e6246b183293412494919f1e7efc9448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:46.951406 29315 ts_tablet_manager.cc:1398] T 272eee3450cf4f48a4e1de2dcb28c0f7 P e6246b183293412494919f1e7efc9448: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:46.951926 19527 tablet_copy_service.cc:342] P 99a0be6ef2144b7fb247ba9d3f051d66: Request end of tablet copy session e6246b183293412494919f1e7efc9448-272eee3450cf4f48a4e1de2dcb28c0f7 received from {username='slave'} at 127.7.39.134:39861
I20240501 14:10:46.952004 19527 tablet_copy_service.cc:433] P 99a0be6ef2144b7fb247ba9d3f051d66: ending tablet copy session e6246b183293412494919f1e7efc9448-272eee3450cf4f48a4e1de2dcb28c0f7 on tablet 272eee3450cf4f48a4e1de2dcb28c0f7 with peer e6246b183293412494919f1e7efc9448
I20240501 14:10:46.962256 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:46.972011 19356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:46.972429 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:46.973217 21268 raft_consensus.cc:1203] T f69020bb6c244848a6b0a50fd2b4f3eb P e6246b183293412494919f1e7efc9448 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20240501 14:10:46.982365 29389 raft_consensus.cc:979] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc: : Instructing follower 5bf81f3d986e428190d3f7727f36d96c to start an election
I20240501 14:10:46.982466 29389 raft_consensus.cc:1067] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Signalling peer 5bf81f3d986e428190d3f7727f36d96c to start an election
I20240501 14:10:46.984246 21136 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "98af8b06b9324e3fa7cdb898b7b9fab9"
dest_uuid: "5bf81f3d986e428190d3f7727f36d96c"
from {username='slave'} at 127.7.39.129:53077
I20240501 14:10:46.984457 21136 raft_consensus.cc:482] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:10:46.984534 21136 raft_consensus.cc:3037] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:46.985641 21136 raft_consensus.cc:504] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 5bf81f3d986e428190d3f7727f36d96c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:46.986966 21136 leader_election.cc:288] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 5bf81f3d986e428190d3f7727f36d96c [CANDIDATE]: Term 2 election: Requested vote from peers 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829), fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847), 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135:38751)
I20240501 14:10:46.988335 19506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98af8b06b9324e3fa7cdb898b7b9fab9" candidate_uuid: "5bf81f3d986e428190d3f7727f36d96c" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "99a0be6ef2144b7fb247ba9d3f051d66"
I20240501 14:10:46.988459 19506 raft_consensus.cc:3037] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:46.989590 19506 raft_consensus.cc:2454] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf81f3d986e428190d3f7727f36d96c in term 2.
I20240501 14:10:46.990917 19375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98af8b06b9324e3fa7cdb898b7b9fab9" candidate_uuid: "5bf81f3d986e428190d3f7727f36d96c" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc"
I20240501 14:10:46.991036 19375 raft_consensus.cc:3032] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:10:46.991096 19375 raft_consensus.cc:729] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fb1e6bc2d07b4ce3b622a66f14c93bdc, State: Running, Role: LEADER
I20240501 14:10:46.991232 19375 consensus_queue.cc:260] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:46.991397 19375 raft_consensus.cc:3037] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:46.992803 19375 raft_consensus.cc:2454] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf81f3d986e428190d3f7727f36d96c in term 2.
I20240501 14:10:46.993170 21070 leader_election.cc:302] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 5bf81f3d986e428190d3f7727f36d96c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 5bf81f3d986e428190d3f7727f36d96c, 99a0be6ef2144b7fb247ba9d3f051d66, fb1e6bc2d07b4ce3b622a66f14c93bdc; no voters:
I20240501 14:10:46.993424 28941 raft_consensus.cc:2781] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 5bf81f3d986e428190d3f7727f36d96c [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:10:46.993577 28941 raft_consensus.cc:686] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 5bf81f3d986e428190d3f7727f36d96c [term 2 LEADER]: Becoming Leader. State: Replica: 5bf81f3d986e428190d3f7727f36d96c, State: Running, Role: LEADER
I20240501 14:10:46.993816 19748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:46.993893 28941 consensus_queue.cc:237] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 5bf81f3d986e428190d3f7727f36d96c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:46.996937 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:46.998230 19231 catalog_manager.cc:5484] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 5bf81f3d986e428190d3f7727f36d96c reported cstate change: term changed from 1 to 2, leader changed from fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) to 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133). New cstate: current_term: 2 leader_uuid: "5bf81f3d986e428190d3f7727f36d96c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:47.001344 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:47.007400 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:47.010355 21247 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:47.020550 21247 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:47.034858 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:47.034864 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:47.049118 21247 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:47.052371 19748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:47.063609 21117 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:47.069367 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:47.095247 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:47.096743 29315 ts_tablet_manager.cc:902] T 410a7dd0dd734684a82875912966657d P e6246b183293412494919f1e7efc9448: Initiating tablet copy from peer fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847)
F20240501 14:10:47.096943 29315 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-11f64b59f83e458f8cf51fd5bcd36fee.queue_time_us, tablet-download-pool-11f64b59f83e458f8cf51fd5bcd36fee.queue_time_us) (tablet-download-pool-11f64b59f83e458f8cf51fd5bcd36fee.run_cpu_time_us, tablet-download-pool-11f64b59f83e458f8cf51fd5bcd36fee.run_cpu_time_us) (tablet-download-pool-11f64b59f83e458f8cf51fd5bcd36fee.run_wall_time_us, tablet-download-pool-11f64b59f83e458f8cf51fd5bcd36fee.run_wall_time_us) (tablet-download-pool-12ffa74f7a5b472aac57253693273cad.queue_time_us, tablet-download-pool-12ffa74f7a5b472aac57253693273cad.queue_time_us) (tablet-download-pool-12ffa74f7a5b472aac57253693273cad.run_cpu_time_us, tablet-download-pool-12ffa74f7a5b472aac57253693273cad.run_cpu_time_us) (tablet-download-pool-12ffa74f7a5b472aac57253693273cad.run_wall_time_us, tablet-download-pool-12ffa74f7a5b472aac57253693273cad.run_wall_time_us) (tablet-download-pool-272eee3450cf4f48a4e1de2dcb28c0f7.queue_time_us, tablet-download-pool-272eee3450cf4f48a4e1de2dcb28c0f7.queue_time_us) (tablet-download-pool-272eee3450cf4f48a4e1de2dcb28c0f7.run_cpu_time_us, tablet-download-pool-272eee3450cf4f48a4e1de2dcb28c0f7.run_cpu_time_us) (tablet-download-pool-272eee3450cf4f48a4e1de2dcb28c0f7.run_wall_time_us, tablet-download-pool-272eee3450cf4f48a4e1de2dcb28c0f7.run_wall_time_us) (tablet-download-pool-2dcca6e0e6374520bd4522c80c110e97.queue_time_us, tablet-download-pool-2dcca6e0e6374520bd4522c80c110e97.queue_time_us) (tablet-download-pool-2dcca6e0e6374520bd4522c80c110e97.run_cpu_time_us, tablet-download-pool-2dcca6e0e6374520bd4522c80c110e97.run_cpu_time_us) (tablet-download-pool-2dcca6e0e6374520bd4522c80c110e97.run_wall_time_us, tablet-download-pool-2dcca6e0e6374520bd4522c80c110e97.run_wall_time_us) (tablet-download-pool-410a7dd0dd734684a82875912966657d.queue_time_us, tablet-download-pool-410a7dd0dd734684a82875912966657d.queue_time_us) (tablet-download-pool-661aaf83e3474966a257fcc039ef4784.queue_time_us, tablet-download-pool-661aaf83e3474966a257fcc039ef4784.queue_time_us) (tablet-download-pool-661aaf83e3474966a257fcc039ef4784.run_cpu_time_us, tablet-download-pool-661aaf83e3474966a257fcc039ef4784.run_cpu_time_us) (tablet-download-pool-661aaf83e3474966a257fcc039ef4784.run_wall_time_us, tablet-download-pool-661aaf83e3474966a257fcc039ef4784.run_wall_time_us) (tablet-download-pool-814ba2247a314a4797aa0f9f624aaeb6.queue_time_us, tablet-download-pool-814ba2247a314a4797aa0f9f624aaeb6.queue_time_us) (tablet-download-pool-814ba2247a314a4797aa0f9f624aaeb6.run_cpu_time_us, tablet-download-pool-814ba2247a314a4797aa0f9f624aaeb6.run_cpu_time_us) (tablet-download-pool-814ba2247a314a4797aa0f9f624aaeb6.run_wall_time_us, tablet-download-pool-814ba2247a314a4797aa0f9f624aaeb6.run_wall_time_us) (tablet-download-pool-85593473500b4d5eaf9f8b0bf2296777.queue_time_us, tablet-download-pool-85593473500b4d5eaf9f8b0bf2296777.queue_time_us) (tablet-download-pool-85593473500b4d5eaf9f8b0bf2296777.run_cpu_time_us, tablet-download-pool-85593473500b4d5eaf9f8b0bf2296777.run_cpu_time_us) (tablet-download-pool-85593473500b4d5eaf9f8b0bf2296777.run_wall_time_us, tablet-download-pool-85593473500b4d5eaf9f8b0bf2296777.run_wall_time_us) (tablet-download-pool-ad7c255baee74e629e2976a942a4174f.queue_time_us, tablet-download-pool-ad7c255baee74e629e2976a942a4174f.queue_time_us) (tablet-download-pool-ad7c255baee74e629e2976a942a4174f.run_cpu_time_us, tablet-download-pool-ad7c255baee74e629e2976a942a4174f.run_cpu_time_us) (tablet-download-pool-ad7c255baee74e629e2976a942a4174f.run_wall_time_us, tablet-download-pool-ad7c255baee74e629e2976a942a4174f.run_wall_time_us) (tablet-download-pool-cc8e888d46d34a22b6d4409ec2ab53a9.queue_time_us, tablet-download-pool-cc8e888d46d34a22b6d4409ec2ab53a9.queue_time_us) (tablet-download-pool-cc8e888d46d34a22b6d4409ec2ab53a9.run_cpu_time_us, tablet-download-pool-cc8e888d46d34a22b6d4409ec2ab53a9.run_cpu_time_us) (tablet-download-pool-cc8e888d46d34a22b6d4409ec2ab53a9.run_wall_time_us, tablet-download-pool-cc8e888d46d34a22b6d4409ec2ab53a9.run_wall_time_us) (tablet-download-pool-d63afd081986473aa2715d074b123a93.queue_time_us, tablet-download-pool-d63afd081986473aa2715d074b123a93.queue_time_us) (tablet-download-pool-d63afd081986473aa2715d074b123a93.run_cpu_time_us, tablet-download-pool-d63afd081986473aa2715d074b123a93.run_cpu_time_us) (tablet-download-pool-d63afd081986473aa2715d074b123a93.run_wall_time_us, tablet-download-pool-d63afd081986473aa2715d074b123a93.run_wall_time_us) (tablet-download-pool-db0f01ee72b24ec9b84410cf94ef9759.queue_time_us, tablet-download-pool-db0f01ee72b24ec9b84410cf94ef9759.queue_time_us) (tablet-download-pool-db0f01ee72b24ec9b84410cf94ef9759.run_cpu_time_us, tablet-download-pool-db0f01ee72b24ec9b84410cf94ef9759.run_cpu_time_us) (tablet-download-pool-db0f01ee72b24ec9b84410cf94ef9759.run_wall_time_us, tablet-download-pool-db0f01ee72b24ec9b84410cf94ef9759.run_wall_time_us) (tablet-download-pool-f69020bb6c244848a6b0a50fd2b4f3eb.queue_time_us, tablet-download-pool-f69020bb6c244848a6b0a50fd2b4f3eb.queue_time_us) (tablet-download-pool-f69020bb6c244848a6b0a50fd2b4f3eb.run_cpu_time_us, tablet-download-pool-f69020bb6c244848a6b0a50fd2b4f3eb.run_cpu_time_us) (tablet-download-pool-f69020bb6c244848a6b0a50fd2b4f3eb.run_wall_time_us, tablet-download-pool-f69020bb6c244848a6b0a50fd2b4f3eb.run_wall_time_us) (tablet-download-pool-fe554c3145de46cf8070efcecffbaedb.queue_time_us, tablet-download-pool-fe554c3145de46cf8070efcecffbaedb.queue_time_us) (tablet-download-pool-fe554c3145de46cf8070efcecffbaedb.run_cpu_time_us, tablet-download-pool-fe554c3145de46cf8070efcecffbaedb.run_cpu_time_us) (tablet-download-pool-fe554c3145de46cf8070efcecffbaedb.run_wall_time_us, tablet-download-pool-fe554c3145de46cf8070efcecffbaedb.run_wall_time_us) (tablet-download-pool-ffb4b6a2c19b497f8117fbbc9ea50082.queue_time_us, tablet-download-pool-ffb4b6a2c19b497f8117fbbc9ea50082.queue_time_us) (tablet-download-pool-ffb4b6a2c19b497f8117fbbc9ea50082.run_cpu_time_us, tablet-download-pool-ffb4b6a2c19b497f8117fbbc9ea50082.run_cpu_time_us) (tablet-download-pool-ffb4b6a2c19b497f8117fbbc9ea50082.run_wall_time_us, tablet-download-pool-ffb4b6a2c19b497f8117fbbc9ea50082.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
I20240501 14:10:47.097218 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
*** Check failure stack trace: ***
I20240501 14:10:47.097537 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
@ 0x7fc65af95dcd google::LogMessage::Fail() at ??:0
@ 0x7fc65af99b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fc65af957cc google::LogMessage::Flush() at ??:0
@ 0x7fc65af96f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc65c3ff044 kudu::TraceMetrics::InternName() at ??:0
I20240501 14:10:47.121428 21117 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
@ 0x7fc65c3e3f98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7fc65c3e1efc kudu::ThreadPoolBuilder::Build() at ??:0
I20240501 14:10:47.139101 21399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98af8b06b9324e3fa7cdb898b7b9fab9" candidate_uuid: "5bf81f3d986e428190d3f7727f36d96c" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "43ce4a4c954f4a5f80954b38933c189d"
I20240501 14:10:47.139250 21399 raft_consensus.cc:3037] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:47.140374 21399 raft_consensus.cc:2454] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 43ce4a4c954f4a5f80954b38933c189d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf81f3d986e428190d3f7727f36d96c in term 2.
I20240501 14:10:47.145099 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:47.147119 19769 raft_consensus.cc:1261] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4 [term 2 FOLLOWER]: Refusing update from remote peer 778c78105092428bbcde75ab2cdb6a05: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:10:47.147606 29284 consensus_queue.cc:1035] T ec2a46f8cfb3406082970e5fdb59b101 P 778c78105092428bbcde75ab2cdb6a05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
@ 0x7fc65ef9477a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7fc65ef9da45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240501 14:10:47.186101 21137 raft_consensus.cc:1261] T ec2a46f8cfb3406082970e5fdb59b101 P 5bf81f3d986e428190d3f7727f36d96c [term 2 FOLLOWER]: Refusing update from remote peer 778c78105092428bbcde75ab2cdb6a05: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:10:47.186545 29284 consensus_queue.cc:1035] T ec2a46f8cfb3406082970e5fdb59b101 P 778c78105092428bbcde75ab2cdb6a05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
@ 0x7fc65f01e76f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240501 14:10:47.196818 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
@ 0x7fc65f0324f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240501 14:10:47.205413 19375 raft_consensus.cc:1261] T ec2a46f8cfb3406082970e5fdb59b101 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 2 FOLLOWER]: Refusing update from remote peer 778c78105092428bbcde75ab2cdb6a05: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:10:47.206252 29415 consensus_queue.cc:1035] T ec2a46f8cfb3406082970e5fdb59b101 P 778c78105092428bbcde75ab2cdb6a05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
@ 0x7fc65f01d226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7fc65f02cc0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20240501 14:10:47.217761 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:47.222000 21117 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:47.224362 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:47.231415 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:47.235186 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:47.239310 21268 raft_consensus.cc:1203] T 272eee3450cf4f48a4e1de2dcb28c0f7 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:10:47.243631 21247 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:47.275426 19748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
@ 0x561285bd275c std::function<>::operator()() at ??:0
@ 0x7fc65c3e6e06 kudu::ThreadPool::DispatchThread() at ??:0
I20240501 14:10:47.309727 19375 consensus_queue.cc:237] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:47.311486 21268 raft_consensus.cc:1261] T d63afd081986473aa2715d074b123a93 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
@ 0x7fc65c3e76ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
I20240501 14:10:47.312122 29391 consensus_queue.cc:1035] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:10:47.319522 28229 raft_consensus.cc:2932] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:47.321992 21268 raft_consensus.cc:2932] T d63afd081986473aa2715d074b123a93 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:47.322589 19231 catalog_manager.cc:5484] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc reported cstate change: config changed from index 7 to 8, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New cstate: current_term: 1 leader_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
@ 0x7fc65c3e903b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20240501 14:10:47.326556 19375 consensus_queue.cc:237] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
@ 0x561285bd275c std::function<>::operator()() at ??:0
I20240501 14:10:47.327302 19485 tablet_service.cc:1514] Processing DeleteTablet for tablet d63afd081986473aa2715d074b123a93 with delete_type TABLET_DATA_TOMBSTONED (TS 99a0be6ef2144b7fb247ba9d3f051d66 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:37600
I20240501 14:10:47.329495 21268 raft_consensus.cc:1261] T d63afd081986473aa2715d074b123a93 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
@ 0x7fc65c3d6807 kudu::Thread::SuperviseThread() at ??:0
W20240501 14:10:47.330116 19309 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135:38751): Couldn't send request to peer 43ce4a4c954f4a5f80954b38933c189d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d63afd081986473aa2715d074b123a93. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:47.330308 29389 consensus_queue.cc:1035] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:10:47.332486 29670 tablet_replica.cc:330] T d63afd081986473aa2715d074b123a93 P 99a0be6ef2144b7fb247ba9d3f051d66: stopping tablet replica
I20240501 14:10:47.332419 29389 raft_consensus.cc:2932] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:47.332742 29670 raft_consensus.cc:2229] T d63afd081986473aa2715d074b123a93 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:47.332865 29670 raft_consensus.cc:2258] T d63afd081986473aa2715d074b123a93 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:47.333374 29670 ts_tablet_manager.cc:1875] T d63afd081986473aa2715d074b123a93 P 99a0be6ef2144b7fb247ba9d3f051d66: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:47.334196 19218 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet d63afd081986473aa2715d074b123a93 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:10:47.334997 19230 catalog_manager.cc:5484] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc reported cstate change: config changed from index 8 to 9, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New cstate: current_term: 1 leader_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
@ 0x7fc65cc146db start_thread at ??:0
I20240501 14:10:47.337442 29670 ts_tablet_manager.cc:1888] T d63afd081986473aa2715d074b123a93 P 99a0be6ef2144b7fb247ba9d3f051d66: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 14:10:47.337513 29670 log.cc:1196] T d63afd081986473aa2715d074b123a93 P 99a0be6ef2144b7fb247ba9d3f051d66: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal/wals/d63afd081986473aa2715d074b123a93
@ 0x7fc65a14c71f clone at ??:0
I20240501 14:10:47.337698 21266 raft_consensus.cc:2932] T d63afd081986473aa2715d074b123a93 P e6246b183293412494919f1e7efc9448 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:10:47.339694 19219 catalog_manager.cc:4830] TS 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829): tablet d63afd081986473aa2715d074b123a93 (table rebalance_test_table_6 [id=afc36d383e3e4795ad11804e9bebcb51]) successfully deleted
I20240501 14:10:47.404914 19637 consensus_queue.cc:237] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:47.407351 21399 raft_consensus.cc:1261] T db0f01ee72b24ec9b84410cf94ef9759 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240501 14:10:47.407852 19573 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357): Couldn't send request to peer da3c4e948ff747678f18456c134d7ed4. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:47.408300 29109 consensus_queue.cc:1035] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:10:47.464370 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:47.471280 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:47.473325 19748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:47.490597 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:47.507894 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:47.510375 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:47.523872 21117 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:47.527930 19375 raft_consensus.cc:1261] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 2 FOLLOWER]: Refusing update from remote peer 5bf81f3d986e428190d3f7727f36d96c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:10:47.528481 29574 consensus_queue.cc:1035] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 5bf81f3d986e428190d3f7727f36d96c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:10:47.556980 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:47.574476 19506 raft_consensus.cc:1261] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 2 FOLLOWER]: Refusing update from remote peer 5bf81f3d986e428190d3f7727f36d96c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:10:47.574950 29574 consensus_queue.cc:1035] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 5bf81f3d986e428190d3f7727f36d96c [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:10:47.587448 21117 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:47.591071 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:47.595220 21399 raft_consensus.cc:1261] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 43ce4a4c954f4a5f80954b38933c189d [term 2 FOLLOWER]: Refusing update from remote peer 5bf81f3d986e428190d3f7727f36d96c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:10:47.595672 29574 consensus_queue.cc:1035] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 5bf81f3d986e428190d3f7727f36d96c [LEADER]: Connected to new peer: Peer: permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:10:47.607252 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:47.630489 19748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:47.678695 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:47.700827 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:47.728597 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:47.744908 21117 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:47.746003 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:10:47.764735 19441 connection.cc:535] client connection to 127.7.39.134:41911 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:10:47.764907 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:10:47.765070 21333 connection.cc:535] client connection to 127.7.39.134:41911 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:10:47.765206 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:10:47.765216 19572 connection.cc:535] client connection to 127.7.39.134:41911 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:10:47.765259 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.39.134:41911 (error 108)
W20240501 14:10:47.765311 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:10:47.774988 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:47.775946 19748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:47.778393 29860 ts_tablet_manager.cc:902] T d63afd081986473aa2715d074b123a93 P 43ce4a4c954f4a5f80954b38933c189d: Initiating tablet copy from peer fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847)
I20240501 14:10:47.778882 29860 tablet_copy_client.cc:308] T d63afd081986473aa2715d074b123a93 P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.129:35847
I20240501 14:10:47.779387 19396 tablet_copy_service.cc:140] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Received BeginTabletCopySession request for tablet d63afd081986473aa2715d074b123a93 from peer 43ce4a4c954f4a5f80954b38933c189d ({username='slave'} at 127.7.39.135:49533)
I20240501 14:10:47.779501 19396 tablet_copy_service.cc:161] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Beginning new tablet copy session on tablet d63afd081986473aa2715d074b123a93 from peer 43ce4a4c954f4a5f80954b38933c189d at {username='slave'} at 127.7.39.135:49533: session id = 43ce4a4c954f4a5f80954b38933c189d-d63afd081986473aa2715d074b123a93
I20240501 14:10:47.780599 19396 tablet_copy_source_session.cc:205] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:47.782038 29860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d63afd081986473aa2715d074b123a93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:47.784507 29860 tablet_copy_client.cc:783] T d63afd081986473aa2715d074b123a93 P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:47.784658 29860 tablet_copy_client.cc:647] T d63afd081986473aa2715d074b123a93 P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:47.784970 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:47.785828 29860 tablet_copy_client.cc:523] T d63afd081986473aa2715d074b123a93 P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:47.787182 29860 tablet_bootstrap.cc:492] T d63afd081986473aa2715d074b123a93 P 43ce4a4c954f4a5f80954b38933c189d: Bootstrap starting.
I20240501 14:10:47.791116 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:10:47.810176 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:47.815449 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:10:47.815552 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:47.817322 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:47.817430 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:47.817466 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:47.817528 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:47.819360 29860 tablet_bootstrap.cc:492] T d63afd081986473aa2715d074b123a93 P 43ce4a4c954f4a5f80954b38933c189d: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:47.819662 29860 tablet_bootstrap.cc:492] T d63afd081986473aa2715d074b123a93 P 43ce4a4c954f4a5f80954b38933c189d: Bootstrap complete.
I20240501 14:10:47.819835 29860 ts_tablet_manager.cc:1367] T d63afd081986473aa2715d074b123a93 P 43ce4a4c954f4a5f80954b38933c189d: Time spent bootstrapping tablet: real 0.033s user 0.022s sys 0.006s
I20240501 14:10:47.820267 29860 raft_consensus.cc:348] T d63afd081986473aa2715d074b123a93 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:47.820377 29860 raft_consensus.cc:729] T d63afd081986473aa2715d074b123a93 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43ce4a4c954f4a5f80954b38933c189d, State: Initialized, Role: FOLLOWER
I20240501 14:10:47.820521 29860 consensus_queue.cc:260] T d63afd081986473aa2715d074b123a93 P 43ce4a4c954f4a5f80954b38933c189d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:47.820865 29860 ts_tablet_manager.cc:1398] T d63afd081986473aa2715d074b123a93 P 43ce4a4c954f4a5f80954b38933c189d: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240501 14:10:47.821467 19396 tablet_copy_service.cc:342] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Request end of tablet copy session 43ce4a4c954f4a5f80954b38933c189d-d63afd081986473aa2715d074b123a93 received from {username='slave'} at 127.7.39.135:49533
I20240501 14:10:47.821563 19396 tablet_copy_service.cc:433] P fb1e6bc2d07b4ce3b622a66f14c93bdc: ending tablet copy session 43ce4a4c954f4a5f80954b38933c189d-d63afd081986473aa2715d074b123a93 on tablet d63afd081986473aa2715d074b123a93 with peer 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:47.823711 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:47.824353 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:47.829056 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:47.833532 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:47.838217 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:47.860666 19769 consensus_queue.cc:237] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:47.861898 21137 raft_consensus.cc:1261] T 1d408b075d784953a57554bcdf19158a P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:10:47.862588 29628 consensus_queue.cc:1035] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:10:47.864538 29901 raft_consensus.cc:2932] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:47.864887 21137 raft_consensus.cc:2932] T 1d408b075d784953a57554bcdf19158a P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:47.868005 19229 catalog_manager.cc:5484] T 1d408b075d784953a57554bcdf19158a P 5bf81f3d986e428190d3f7727f36d96c reported cstate change: config changed from index 7 to 8, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) evicted. New cstate: current_term: 1 leader_uuid: "da3c4e948ff747678f18456c134d7ed4" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:47.874024 29915 ts_tablet_manager.cc:902] T db0f01ee72b24ec9b84410cf94ef9759 P da3c4e948ff747678f18456c134d7ed4: Initiating tablet copy from peer ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201)
I20240501 14:10:47.874609 29915 tablet_copy_client.cc:308] T db0f01ee72b24ec9b84410cf94ef9759 P da3c4e948ff747678f18456c134d7ed4: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.131:45201
I20240501 14:10:47.875217 19657 tablet_copy_service.cc:140] P ec8cde17a00645e7b38d5d29f7425367: Received BeginTabletCopySession request for tablet db0f01ee72b24ec9b84410cf94ef9759 from peer da3c4e948ff747678f18456c134d7ed4 ({username='slave'} at 127.7.39.132:44167)
I20240501 14:10:47.875330 19657 tablet_copy_service.cc:161] P ec8cde17a00645e7b38d5d29f7425367: Beginning new tablet copy session on tablet db0f01ee72b24ec9b84410cf94ef9759 from peer da3c4e948ff747678f18456c134d7ed4 at {username='slave'} at 127.7.39.132:44167: session id = da3c4e948ff747678f18456c134d7ed4-db0f01ee72b24ec9b84410cf94ef9759
I20240501 14:10:47.876497 19657 tablet_copy_source_session.cc:205] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:47.878072 29915 ts_tablet_manager.cc:1875] T db0f01ee72b24ec9b84410cf94ef9759 P da3c4e948ff747678f18456c134d7ed4: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:10:47.880934 29915 ts_tablet_manager.cc:1888] T db0f01ee72b24ec9b84410cf94ef9759 P da3c4e948ff747678f18456c134d7ed4: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 14:10:47.881212 29915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db0f01ee72b24ec9b84410cf94ef9759. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:47.882769 29915 tablet_copy_client.cc:783] T db0f01ee72b24ec9b84410cf94ef9759 P da3c4e948ff747678f18456c134d7ed4: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:47.883034 29915 tablet_copy_client.cc:647] T db0f01ee72b24ec9b84410cf94ef9759 P da3c4e948ff747678f18456c134d7ed4: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:47.887377 19485 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d408b075d784953a57554bcdf19158a with delete_type TABLET_DATA_TOMBSTONED (TS 99a0be6ef2144b7fb247ba9d3f051d66 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:37600
I20240501 14:10:47.887812 29945 tablet_replica.cc:330] T 1d408b075d784953a57554bcdf19158a P 99a0be6ef2144b7fb247ba9d3f051d66: stopping tablet replica
I20240501 14:10:47.887840 19769 consensus_queue.cc:237] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:10:47.888072 29945 raft_consensus.cc:2229] T 1d408b075d784953a57554bcdf19158a P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:47.888301 29945 raft_consensus.cc:2258] T 1d408b075d784953a57554bcdf19158a P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:47.889530 29915 tablet_copy_client.cc:523] T db0f01ee72b24ec9b84410cf94ef9759 P da3c4e948ff747678f18456c134d7ed4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:47.889611 29945 ts_tablet_manager.cc:1875] T 1d408b075d784953a57554bcdf19158a P 99a0be6ef2144b7fb247ba9d3f051d66: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:47.889998 21137 raft_consensus.cc:1261] T 1d408b075d784953a57554bcdf19158a P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:10:47.890650 29628 consensus_queue.cc:1035] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 14:10:47.891017 19702 consensus_peers.cc:485] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4 -> Peer 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136:34895): Couldn't send request to peer 778c78105092428bbcde75ab2cdb6a05. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1d408b075d784953a57554bcdf19158a. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:47.891271 29915 tablet_bootstrap.cc:492] T db0f01ee72b24ec9b84410cf94ef9759 P da3c4e948ff747678f18456c134d7ed4: Bootstrap starting.
I20240501 14:10:47.893850 29945 ts_tablet_manager.cc:1888] T 1d408b075d784953a57554bcdf19158a P 99a0be6ef2144b7fb247ba9d3f051d66: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 14:10:47.893996 29945 log.cc:1196] T 1d408b075d784953a57554bcdf19158a P 99a0be6ef2144b7fb247ba9d3f051d66: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal/wals/1d408b075d784953a57554bcdf19158a
I20240501 14:10:47.894430 29628 raft_consensus.cc:2932] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:47.894656 19219 catalog_manager.cc:4830] TS 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829): tablet 1d408b075d784953a57554bcdf19158a (table rebalance_test_table_0 [id=b52269ef7c494f74a5e4f0d5cb3b8a47]) successfully deleted
I20240501 14:10:47.895910 19220 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 1d408b075d784953a57554bcdf19158a with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:10:47.896975 21137 raft_consensus.cc:2932] T 1d408b075d784953a57554bcdf19158a P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:47.897475 19229 catalog_manager.cc:5484] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4 reported cstate change: config changed from index 8 to 9, VOTER 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136) added. New cstate: current_term: 1 leader_uuid: "da3c4e948ff747678f18456c134d7ed4" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:47.930096 29915 tablet_bootstrap.cc:492] T db0f01ee72b24ec9b84410cf94ef9759 P da3c4e948ff747678f18456c134d7ed4: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 14:10:47.930572 29915 tablet_bootstrap.cc:492] T db0f01ee72b24ec9b84410cf94ef9759 P da3c4e948ff747678f18456c134d7ed4: Bootstrap complete.
I20240501 14:10:47.930785 29915 ts_tablet_manager.cc:1367] T db0f01ee72b24ec9b84410cf94ef9759 P da3c4e948ff747678f18456c134d7ed4: Time spent bootstrapping tablet: real 0.040s user 0.026s sys 0.002s
I20240501 14:10:47.931401 29915 raft_consensus.cc:348] T db0f01ee72b24ec9b84410cf94ef9759 P da3c4e948ff747678f18456c134d7ed4 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:47.932339 29915 raft_consensus.cc:729] T db0f01ee72b24ec9b84410cf94ef9759 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3c4e948ff747678f18456c134d7ed4, State: Initialized, Role: FOLLOWER
I20240501 14:10:47.932535 29915 consensus_queue.cc:260] T db0f01ee72b24ec9b84410cf94ef9759 P da3c4e948ff747678f18456c134d7ed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:10:47.935269 29915 ts_tablet_manager.cc:1398] T db0f01ee72b24ec9b84410cf94ef9759 P da3c4e948ff747678f18456c134d7ed4: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
W20240501 14:10:47.935930 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:10:47.936542 19657 tablet_copy_service.cc:342] P ec8cde17a00645e7b38d5d29f7425367: Request end of tablet copy session da3c4e948ff747678f18456c134d7ed4-db0f01ee72b24ec9b84410cf94ef9759 received from {username='slave'} at 127.7.39.132:44167
I20240501 14:10:47.936622 19657 tablet_copy_service.cc:433] P ec8cde17a00645e7b38d5d29f7425367: ending tablet copy session da3c4e948ff747678f18456c134d7ed4-db0f01ee72b24ec9b84410cf94ef9759 on tablet db0f01ee72b24ec9b84410cf94ef9759 with peer da3c4e948ff747678f18456c134d7ed4
W20240501 14:10:47.936998 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:47.970726 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:47.982057 21117 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:47.991197 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:47.995380 19748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:48.011797 21117 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:48.016590 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:48.025727 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:48.027558 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:48.031935 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:48.047541 19748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:48.048955 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:48.070112 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:48.071121 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:48.093286 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:48.095361 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:48.104183 19748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:48.105264 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:48.106020 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:48.106698 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:48.116209 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:48.134214 21117 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:48.175329 21582 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240501 14:10:32.527823 21587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:10:32.528193 21587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:10:32.533052 21587 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:10:33.614869 21587 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 14:10:33.852120 22157 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTabletLocations -> {remote=127.7.39.190:34571, user_credentials={real_user=slave}} blocked reactor thread for 45461us
W20240501 14:10:34.170588 21587 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1fbbd460d13647c1a85b517926267158: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server e6246b183293412494919f1e7efc9448
W20240501 14:10:34.170799 21587 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b52269ef7c494f74a5e4f0d5cb3b8a47: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server e6246b183293412494919f1e7efc9448
W20240501 14:10:34.170912 21587 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3866c35dd0e64d87b6049e5e14cb0d31: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:34.171299 21587 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a9b1c37e923e49a7b30905624977117c: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:34.171401 21587 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1fbbd460d13647c1a85b517926267158: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:34.171494 21587 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afc36d383e3e4795ad11804e9bebcb51: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:34.171572 21587 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 151dfad0ef40470ab9c5203c42400b60: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server e6246b183293412494919f1e7efc9448
W20240501 14:10:34.171680 21587 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6f6b0a378a484e5889d733ff982d9eb7: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:34.171772 21587 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4d7621338643426d90fe03eb71654cb8: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:34.171823 21587 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afc36d383e3e4795ad11804e9bebcb51: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server e6246b183293412494919f1e7efc9448
W20240501 14:10:34.171943 21587 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ed1a7ed87fa9480ba18a052130ecdc05: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:34.172050 21587 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 35d6f8fac5394197b987aa44dab206c1: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server e6246b183293412494919f1e7efc9448
W20240501 14:10:34.172102 21587 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a9b1c37e923e49a7b30905624977117c: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:34.172154 21587 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6f6b0a378a484e5889d733ff982d9eb7: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:34.172195 21587 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4d7621338643426d90fe03eb71654cb8: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 5bf81f3d986e428190d3f7727f36d96c
I20240501 14:10:34.259531 21587 rebalancer_tool.cc:1074] tablet db0f01ee72b24ec9b84410cf94ef9759: 'fb1e6bc2d07b4ce3b622a66f14c93bdc' -> 'e6246b183293412494919f1e7efc9448' move scheduled
I20240501 14:10:34.335567 21587 rebalancer_tool.cc:1074] tablet 98af8b06b9324e3fa7cdb898b7b9fab9: 'ec8cde17a00645e7b38d5d29f7425367' -> '43ce4a4c954f4a5f80954b38933c189d' move scheduled
I20240501 14:10:34.389319 21587 rebalancer_tool.cc:1084] tablet cf62635e9a994259b9e293e5eb425b55: '99a0be6ef2144b7fb247ba9d3f051d66' -> '778c78105092428bbcde75ab2cdb6a05' move ignored: Illegal state: tablet cf62635e9a994259b9e293e5eb425b55: replica 778c78105092428bbcde75ab2cdb6a05 is already present
I20240501 14:10:34.389454 21587 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:10:34.944818 21587 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 151dfad0ef40470ab9c5203c42400b60: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:34.945083 21587 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afc36d383e3e4795ad11804e9bebcb51: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:34.945147 21587 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1fbbd460d13647c1a85b517926267158: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server e6246b183293412494919f1e7efc9448
W20240501 14:10:34.945220 21587 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b52269ef7c494f74a5e4f0d5cb3b8a47: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:34.945255 21587 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ed1a7ed87fa9480ba18a052130ecdc05: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:34.945288 21587 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 35d6f8fac5394197b987aa44dab206c1: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:34.945381 21587 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3866c35dd0e64d87b6049e5e14cb0d31: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server e6246b183293412494919f1e7efc9448
W20240501 14:10:34.945418 21587 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b52269ef7c494f74a5e4f0d5cb3b8a47: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:34.945451 21587 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ed1a7ed87fa9480ba18a052130ecdc05: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:34.945530 21587 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4d7621338643426d90fe03eb71654cb8: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:34.958580 21587 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a9b1c37e923e49a7b30905624977117c: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:34.958658 21587 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3866c35dd0e64d87b6049e5e14cb0d31: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:34.958700 21587 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 151dfad0ef40470ab9c5203c42400b60: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:34.958741 21587 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afc36d383e3e4795ad11804e9bebcb51: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:34.958786 21587 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1fbbd460d13647c1a85b517926267158: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:34.958838 21587 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6f6b0a378a484e5889d733ff982d9eb7: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server e6246b183293412494919f1e7efc9448
W20240501 14:10:34.958890 21587 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 35d6f8fac5394197b987aa44dab206c1: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:34.958947 21587 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b52269ef7c494f74a5e4f0d5cb3b8a47: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:34.959038 21587 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4d7621338643426d90fe03eb71654cb8: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:34.959113 21587 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a9b1c37e923e49a7b30905624977117c: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server e6246b183293412494919f1e7efc9448
I20240501 14:10:35.230561 21587 rebalancer_tool.cc:1074] tablet 410a7dd0dd734684a82875912966657d: 'da3c4e948ff747678f18456c134d7ed4' -> '778c78105092428bbcde75ab2cdb6a05' move scheduled
I20240501 14:10:35.525193 21587 rebalancer_tool.cc:1074] tablet 1d408b075d784953a57554bcdf19158a: 'fb1e6bc2d07b4ce3b622a66f14c93bdc' -> '5bf81f3d986e428190d3f7727f36d96c' move scheduled
I20240501 14:10:35.633257 21587 rebalancer_tool.cc:1074] tablet d63afd081986473aa2715d074b123a93: '99a0be6ef2144b7fb247ba9d3f051d66' -> 'e6246b183293412494919f1e7efc9448' move scheduled
I20240501 14:10:35.772419 21587 rebalancer_tool.cc:1074] tablet ec2a46f8cfb3406082970e5fdb59b101: 'ec8cde17a00645e7b38d5d29f7425367' -> '778c78105092428bbcde75ab2cdb6a05' move scheduled
I20240501 14:10:35.864403 21587 rebalancer_tool.cc:1074] tablet f6ca088ce3aa4b97aa54c8ce6d700921: 'da3c4e948ff747678f18456c134d7ed4' -> '43ce4a4c954f4a5f80954b38933c189d' move scheduled
I20240501 14:10:35.953197 21587 rebalancer_tool.cc:1074] tablet bb291a0c743c4e638f79da36cc77faa3: 'fb1e6bc2d07b4ce3b622a66f14c93bdc' -> '5bf81f3d986e428190d3f7727f36d96c' move scheduled
I20240501 14:10:36.259745 21587 rebalancer_tool.cc:1074] tablet 661aaf83e3474966a257fcc039ef4784: '99a0be6ef2144b7fb247ba9d3f051d66' -> 'e6246b183293412494919f1e7efc9448' move scheduled
I20240501 14:10:36.468571 21587 rebalancer_tool.cc:1074] tablet dab5e5722e2e4b0eaea73288d8dce265: 'ec8cde17a00645e7b38d5d29f7425367' -> '5bf81f3d986e428190d3f7727f36d96c' move scheduled
I20240501 14:10:36.541674 21587 rebalancer_tool.cc:1074] tablet fe554c3145de46cf8070efcecffbaedb: '99a0be6ef2144b7fb247ba9d3f051d66' -> '778c78105092428bbcde75ab2cdb6a05' move scheduled
I20240501 14:10:36.568132 21587 rebalancer_tool.cc:1084] tablet 78e14dea8b6e4c0abd1c88dcd5b4e608: 'da3c4e948ff747678f18456c134d7ed4' -> '5bf81f3d986e428190d3f7727f36d96c' move ignored: Illegal state: tablet 78e14dea8b6e4c0abd1c88dcd5b4e608: replica 5bf81f3d986e428190d3f7727f36d96c is already present
I20240501 14:10:36.568246 21587 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:10:37.020967 21587 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a9b1c37e923e49a7b30905624977117c: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:37.021080 21587 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3866c35dd0e64d87b6049e5e14cb0d31: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:37.021128 21587 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6f6b0a378a484e5889d733ff982d9eb7: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:37.021167 21587 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4d7621338643426d90fe03eb71654cb8: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server e6246b183293412494919f1e7efc9448
W20240501 14:10:37.021277 21587 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a9b1c37e923e49a7b30905624977117c: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:37.021350 21587 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 35d6f8fac5394197b987aa44dab206c1: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server 778c78105092428bbcde75ab2cdb6a05
I20240501 14:10:37.155323 21587 rebalancer_tool.cc:1074] tablet ad7c255baee74e629e2976a942a4174f: 'da3c4e948ff747678f18456c134d7ed4' -> '' move scheduled
I20240501 14:10:37.202993 21587 rebalancer_tool.cc:1074] tablet cf62635e9a994259b9e293e5eb425b55: '99a0be6ef2144b7fb247ba9d3f051d66' -> '' move scheduled
I20240501 14:10:37.223691 21587 rebalancer_tool.cc:1074] tablet 2dcca6e0e6374520bd4522c80c110e97: 'fb1e6bc2d07b4ce3b622a66f14c93bdc' -> '' move scheduled
I20240501 14:10:37.235463 21587 rebalancer_tool.cc:1074] tablet 55830424a98348e382759c54afdeabcd: 'ec8cde17a00645e7b38d5d29f7425367' -> '' move scheduled
I20240501 14:10:37.290926 21587 rebalancer_tool.cc:1084] tablet ffb4b6a2c19b497f8117fbbc9ea50082: '99a0be6ef2144b7fb247ba9d3f051d66' -> 'e6246b183293412494919f1e7efc9448' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240501 14:10:37.291010 21587 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:10:37.796304 21587 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a9b1c37e923e49a7b30905624977117c: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:37.798743 21587 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afc36d383e3e4795ad11804e9bebcb51: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:37.798931 21587 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3866c35dd0e64d87b6049e5e14cb0d31: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:37.798993 21587 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6f6b0a378a484e5889d733ff982d9eb7: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:37.799068 21587 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4d7621338643426d90fe03eb71654cb8: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server e6246b183293412494919f1e7efc9448
W20240501 14:10:37.799121 21587 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 35d6f8fac5394197b987aa44dab206c1: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:37.799170 21587 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b52269ef7c494f74a5e4f0d5cb3b8a47: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:37.799212 21587 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a9b1c37e923e49a7b30905624977117c: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:37.799260 21587 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afc36d383e3e4795ad11804e9bebcb51: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server e6246b183293412494919f1e7efc9448
W20240501 14:10:37.799335 21587 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3866c35dd0e64d87b6049e5e14cb0d31: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 778c78105092428bbcde75ab2cdb6a05
I20240501 14:10:37.828827 21587 rebalancer_tool.cc:1074] tablet 85593473500b4d5eaf9f8b0bf2296777: 'da3c4e948ff747678f18456c134d7ed4' -> '' move scheduled
I20240501 14:10:44.556824 21587 rebalancer_tool.cc:1136] tablet 98af8b06b9324e3fa7cdb898b7b9fab9: 'ec8cde17a00645e7b38d5d29f7425367' -> '43ce4a4c954f4a5f80954b38933c189d' move completed: OK
I20240501 14:10:45.776043 21587 rebalancer_tool.cc:1136] tablet ec2a46f8cfb3406082970e5fdb59b101: 'ec8cde17a00645e7b38d5d29f7425367' -> '778c78105092428bbcde75ab2cdb6a05' move completed: OK
I20240501 14:10:45.792567 21587 rebalancer_tool.cc:1136] tablet 85593473500b4d5eaf9f8b0bf2296777: 'da3c4e948ff747678f18456c134d7ed4' -> '' move completed: OK
I20240501 14:10:45.801036 21587 rebalancer_tool.cc:1136] tablet cf62635e9a994259b9e293e5eb425b55: '99a0be6ef2144b7fb247ba9d3f051d66' -> '' move completed: OK
I20240501 14:10:45.812300 21587 rebalancer_tool.cc:1136] tablet 2dcca6e0e6374520bd4522c80c110e97: 'fb1e6bc2d07b4ce3b622a66f14c93bdc' -> '' move completed: OK
I20240501 14:10:45.824297 21587 rebalancer_tool.cc:1136] tablet ad7c255baee74e629e2976a942a4174f: 'da3c4e948ff747678f18456c134d7ed4' -> '' move completed: OK
I20240501 14:10:45.830549 21587 rebalancer_tool.cc:1136] tablet dab5e5722e2e4b0eaea73288d8dce265: 'ec8cde17a00645e7b38d5d29f7425367' -> '5bf81f3d986e428190d3f7727f36d96c' move completed: OK
I20240501 14:10:45.860680 21587 rebalancer_tool.cc:1136] tablet 661aaf83e3474966a257fcc039ef4784: '99a0be6ef2144b7fb247ba9d3f051d66' -> 'e6246b183293412494919f1e7efc9448' move completed: OK
I20240501 14:10:45.870816 21587 rebalancer_tool.cc:1136] tablet fe554c3145de46cf8070efcecffbaedb: '99a0be6ef2144b7fb247ba9d3f051d66' -> '778c78105092428bbcde75ab2cdb6a05' move completed: OK
I20240501 14:10:45.900748 21587 rebalancer_tool.cc:1136] tablet bb291a0c743c4e638f79da36cc77faa3: 'fb1e6bc2d07b4ce3b622a66f14c93bdc' -> '5bf81f3d986e428190d3f7727f36d96c' move completed: OK
I20240501 14:10:45.937706 21587 rebalancer_tool.cc:1136] tablet f6ca088ce3aa4b97aa54c8ce6d700921: 'da3c4e948ff747678f18456c134d7ed4' -> '43ce4a4c954f4a5f80954b38933c189d' move completed: OK
I20240501 14:10:45.962589 21587 rebalancer_tool.cc:1136] tablet db0f01ee72b24ec9b84410cf94ef9759: 'fb1e6bc2d07b4ce3b622a66f14c93bdc' -> 'e6246b183293412494919f1e7efc9448' move completed: OK
I20240501 14:10:46.694801 21587 rebalancer_tool.cc:1136] tablet 1d408b075d784953a57554bcdf19158a: 'fb1e6bc2d07b4ce3b622a66f14c93bdc' -> '5bf81f3d986e428190d3f7727f36d96c' move completed: OK
I20240501 14:10:47.352315 21587 rebalancer_tool.cc:1136] tablet 55830424a98348e382759c54afdeabcd: 'ec8cde17a00645e7b38d5d29f7425367' -> '' move completed: OK
I20240501 14:10:47.371119 21587 rebalancer_tool.cc:1074] tablet 11f64b59f83e458f8cf51fd5bcd36fee: 'fb1e6bc2d07b4ce3b622a66f14c93bdc' -> '' move scheduled
I20240501 14:10:47.396323 21587 rebalancer_tool.cc:1074] tablet 0988c468757d49738765bdcb31794bd1: 'ec8cde17a00645e7b38d5d29f7425367' -> '' move scheduled
I20240501 14:10:47.424018 21587 rebalancer_tool.cc:1074] tablet cc8e888d46d34a22b6d4409ec2ab53a9: '99a0be6ef2144b7fb247ba9d3f051d66' -> '' move scheduled
W20240501 14:10:47.767104 29744 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:10:47.431552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.39.134:41911 (local address 127.0.0.1:54770)
0501 14:10:47.434783 (+ 3231us) negotiation.cc:107] Waiting for socket to connect
0501 14:10:47.434802 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0501 14:10:47.436826 (+ 2024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:10:47.766349 (+329523us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":3134,"thread_start_us":77,"threads_started":1}
I20240501 14:10:47.773252 21587 rebalancer_tool.cc:1084] tablet 814ba2247a314a4797aa0f9f624aaeb6: 'da3c4e948ff747678f18456c134d7ed4' -> '' move ignored: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:10:47.773356 21587 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): unacceptable health status UNAVAILABLE
W20240501 14:10:48.204840 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:10:48.205181 19637 consensus_queue.cc:237] T cf62635e9a994259b9e293e5eb425b55 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:10:48.206501 21530 raft_consensus.cc:1261] T cf62635e9a994259b9e293e5eb425b55 P 778c78105092428bbcde75ab2cdb6a05 [term 2 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:10:48.207021 29109 consensus_queue.cc:1035] T cf62635e9a994259b9e293e5eb425b55 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:10:48.209472 29109 raft_consensus.cc:2932] T cf62635e9a994259b9e293e5eb425b55 P ec8cde17a00645e7b38d5d29f7425367 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:48.209789 21530 raft_consensus.cc:2932] T cf62635e9a994259b9e293e5eb425b55 P 778c78105092428bbcde75ab2cdb6a05 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:48.212842 19229 catalog_manager.cc:5484] T cf62635e9a994259b9e293e5eb425b55 P ec8cde17a00645e7b38d5d29f7425367 reported cstate change: config changed from index 5 to 7, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New cstate: current_term: 2 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:48.216527 19748 tablet_service.cc:1514] Processing DeleteTablet for tablet cf62635e9a994259b9e293e5eb425b55 with delete_type TABLET_DATA_TOMBSTONED (TS da3c4e948ff747678f18456c134d7ed4 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:36530
I20240501 14:10:48.216895 30096 tablet_replica.cc:330] T cf62635e9a994259b9e293e5eb425b55 P da3c4e948ff747678f18456c134d7ed4: stopping tablet replica
I20240501 14:10:48.216944 21399 raft_consensus.cc:1203] T d63afd081986473aa2715d074b123a93 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240501 14:10:48.217518 30096 raft_consensus.cc:2229] T cf62635e9a994259b9e293e5eb425b55 P da3c4e948ff747678f18456c134d7ed4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:48.218222 19637 consensus_queue.cc:237] T cf62635e9a994259b9e293e5eb425b55 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:48.217882 30096 raft_consensus.cc:2258] T cf62635e9a994259b9e293e5eb425b55 P da3c4e948ff747678f18456c134d7ed4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:48.219730 21530 raft_consensus.cc:1261] T cf62635e9a994259b9e293e5eb425b55 P 778c78105092428bbcde75ab2cdb6a05 [term 2 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 14:10:48.219743 30096 ts_tablet_manager.cc:1875] T cf62635e9a994259b9e293e5eb425b55 P da3c4e948ff747678f18456c134d7ed4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:48.220929 29734 consensus_queue.cc:1035] T cf62635e9a994259b9e293e5eb425b55 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:10:48.222424 30096 ts_tablet_manager.cc:1888] T cf62635e9a994259b9e293e5eb425b55 P da3c4e948ff747678f18456c134d7ed4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 14:10:48.222558 30096 log.cc:1196] T cf62635e9a994259b9e293e5eb425b55 P da3c4e948ff747678f18456c134d7ed4: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal/wals/cf62635e9a994259b9e293e5eb425b55
I20240501 14:10:48.223505 19220 catalog_manager.cc:4830] TS da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357): tablet cf62635e9a994259b9e293e5eb425b55 (table rebalance_test_table_4 [id=151dfad0ef40470ab9c5203c42400b60]) successfully deleted
W20240501 14:10:48.225608 19574 consensus_peers.cc:485] T cf62635e9a994259b9e293e5eb425b55 P ec8cde17a00645e7b38d5d29f7425367 -> Peer 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133:46883): Couldn't send request to peer 5bf81f3d986e428190d3f7727f36d96c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cf62635e9a994259b9e293e5eb425b55. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:48.228742 29734 raft_consensus.cc:2932] T cf62635e9a994259b9e293e5eb425b55 P ec8cde17a00645e7b38d5d29f7425367 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:48.229266 21530 raft_consensus.cc:2932] T cf62635e9a994259b9e293e5eb425b55 P 778c78105092428bbcde75ab2cdb6a05 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:48.230724 19219 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet cf62635e9a994259b9e293e5eb425b55 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:10:48.232051 19229 catalog_manager.cc:5484] T cf62635e9a994259b9e293e5eb425b55 P ec8cde17a00645e7b38d5d29f7425367 reported cstate change: config changed from index 7 to 8, VOTER 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133) added. New cstate: current_term: 2 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:48.260906 21117 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:48.261675 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:48.270766 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:48.277626 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:48.283913 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:48.289537 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:48.290167 19748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:48.295857 30135 ts_tablet_manager.cc:902] T 1d408b075d784953a57554bcdf19158a P 778c78105092428bbcde75ab2cdb6a05: Initiating tablet copy from peer da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357)
I20240501 14:10:48.296406 30135 tablet_copy_client.cc:308] T 1d408b075d784953a57554bcdf19158a P 778c78105092428bbcde75ab2cdb6a05: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.132:35357
I20240501 14:10:48.296916 19789 tablet_copy_service.cc:140] P da3c4e948ff747678f18456c134d7ed4: Received BeginTabletCopySession request for tablet 1d408b075d784953a57554bcdf19158a from peer 778c78105092428bbcde75ab2cdb6a05 ({username='slave'} at 127.7.39.136:40943)
I20240501 14:10:48.297024 19789 tablet_copy_service.cc:161] P da3c4e948ff747678f18456c134d7ed4: Beginning new tablet copy session on tablet 1d408b075d784953a57554bcdf19158a from peer 778c78105092428bbcde75ab2cdb6a05 at {username='slave'} at 127.7.39.136:40943: session id = 778c78105092428bbcde75ab2cdb6a05-1d408b075d784953a57554bcdf19158a
I20240501 14:10:48.297960 19789 tablet_copy_source_session.cc:205] T 1d408b075d784953a57554bcdf19158a P da3c4e948ff747678f18456c134d7ed4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:48.298872 30135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d408b075d784953a57554bcdf19158a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:48.301250 30135 tablet_copy_client.cc:783] T 1d408b075d784953a57554bcdf19158a P 778c78105092428bbcde75ab2cdb6a05: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:48.301409 30135 tablet_copy_client.cc:647] T 1d408b075d784953a57554bcdf19158a P 778c78105092428bbcde75ab2cdb6a05: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:48.306399 30135 tablet_copy_client.cc:523] T 1d408b075d784953a57554bcdf19158a P 778c78105092428bbcde75ab2cdb6a05: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:48.307825 30135 tablet_bootstrap.cc:492] T 1d408b075d784953a57554bcdf19158a P 778c78105092428bbcde75ab2cdb6a05: Bootstrap starting.
I20240501 14:10:48.333338 29323 raft_consensus.cc:482] T 814ba2247a314a4797aa0f9f624aaeb6 P 43ce4a4c954f4a5f80954b38933c189d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e6246b183293412494919f1e7efc9448)
I20240501 14:10:48.333494 29323 raft_consensus.cc:504] T 814ba2247a314a4797aa0f9f624aaeb6 P 43ce4a4c954f4a5f80954b38933c189d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:48.333923 29323 leader_election.cc:288] T 814ba2247a314a4797aa0f9f624aaeb6 P 43ce4a4c954f4a5f80954b38933c189d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e6246b183293412494919f1e7efc9448 (127.7.39.134:41911), 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133:46883)
I20240501 14:10:48.335064 21137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "814ba2247a314a4797aa0f9f624aaeb6" candidate_uuid: "43ce4a4c954f4a5f80954b38933c189d" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "5bf81f3d986e428190d3f7727f36d96c" is_pre_election: true
W20240501 14:10:48.337049 21333 leader_election.cc:334] T 814ba2247a314a4797aa0f9f624aaeb6 P 43ce4a4c954f4a5f80954b38933c189d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:10:48.337142 21333 leader_election.cc:302] T 814ba2247a314a4797aa0f9f624aaeb6 P 43ce4a4c954f4a5f80954b38933c189d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43ce4a4c954f4a5f80954b38933c189d; no voters: 5bf81f3d986e428190d3f7727f36d96c, e6246b183293412494919f1e7efc9448
I20240501 14:10:48.338001 29323 raft_consensus.cc:2726] T 814ba2247a314a4797aa0f9f624aaeb6 P 43ce4a4c954f4a5f80954b38933c189d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:10:48.342463 30135 tablet_bootstrap.cc:492] T 1d408b075d784953a57554bcdf19158a P 778c78105092428bbcde75ab2cdb6a05: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:48.342803 30135 tablet_bootstrap.cc:492] T 1d408b075d784953a57554bcdf19158a P 778c78105092428bbcde75ab2cdb6a05: Bootstrap complete.
I20240501 14:10:48.343004 30135 ts_tablet_manager.cc:1367] T 1d408b075d784953a57554bcdf19158a P 778c78105092428bbcde75ab2cdb6a05: Time spent bootstrapping tablet: real 0.035s user 0.020s sys 0.007s
I20240501 14:10:48.343482 30135 raft_consensus.cc:348] T 1d408b075d784953a57554bcdf19158a P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:10:48.343601 30135 raft_consensus.cc:729] T 1d408b075d784953a57554bcdf19158a P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 778c78105092428bbcde75ab2cdb6a05, State: Initialized, Role: FOLLOWER
I20240501 14:10:48.343775 30135 consensus_queue.cc:260] T 1d408b075d784953a57554bcdf19158a P 778c78105092428bbcde75ab2cdb6a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:10:48.344512 30135 ts_tablet_manager.cc:1398] T 1d408b075d784953a57554bcdf19158a P 778c78105092428bbcde75ab2cdb6a05: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:10:48.347501 19789 tablet_copy_service.cc:342] P da3c4e948ff747678f18456c134d7ed4: Request end of tablet copy session 778c78105092428bbcde75ab2cdb6a05-1d408b075d784953a57554bcdf19158a received from {username='slave'} at 127.7.39.136:40943
I20240501 14:10:48.347594 19789 tablet_copy_service.cc:433] P da3c4e948ff747678f18456c134d7ed4: ending tablet copy session 778c78105092428bbcde75ab2cdb6a05-1d408b075d784953a57554bcdf19158a on tablet 1d408b075d784953a57554bcdf19158a with peer 778c78105092428bbcde75ab2cdb6a05
I20240501 14:10:48.379328 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:48.382957 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:48.383173 21117 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:48.390233 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:48.399226 19748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:48.400466 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:48.415459 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:48.445155 19769 raft_consensus.cc:1203] T db0f01ee72b24ec9b84410cf94ef9759 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240501 14:10:48.446712 29109 raft_consensus.cc:2932] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } }
I20240501 14:10:48.448402 29109 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:53950 (request call id 0) took 1044 ms. Trace:
I20240501 14:10:48.448495 29109 rpcz_store.cc:276] 0501 14:10:47.404394 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 14:10:47.404441 (+ 47us) service_pool.cc:225] Handling call
0501 14:10:47.405536 (+ 1095us) log.cc:842] Serialized 482 byte log entry
0501 14:10:48.448388 (+1042852us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":849,"raft.run_cpu_time_us":310,"raft.run_wall_time_us":288,"thread_start_us":137,"threads_started":2,"wal-append.queue_time_us":7997,"wal-append.run_cpu_time_us":344,"wal-append.run_wall_time_us":1000395}
I20240501 14:10:48.453469 19232 catalog_manager.cc:5484] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 reported cstate change: config changed from index 7 to 8, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) added. New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } }
I20240501 14:10:48.453588 21399 raft_consensus.cc:2932] T db0f01ee72b24ec9b84410cf94ef9759 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } }
I20240501 14:10:48.455005 19769 raft_consensus.cc:2932] T db0f01ee72b24ec9b84410cf94ef9759 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } }
I20240501 14:10:48.455734 29806 raft_consensus.cc:482] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e6246b183293412494919f1e7efc9448)
I20240501 14:10:48.455839 29806 raft_consensus.cc:504] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:48.456353 29806 leader_election.cc:288] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e6246b183293412494919f1e7efc9448 (127.7.39.134:41911), 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135:38751)
I20240501 14:10:48.456939 21398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "814ba2247a314a4797aa0f9f624aaeb6" candidate_uuid: "5bf81f3d986e428190d3f7727f36d96c" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "43ce4a4c954f4a5f80954b38933c189d" is_pre_election: true
I20240501 14:10:48.457062 21398 raft_consensus.cc:2454] T 814ba2247a314a4797aa0f9f624aaeb6 P 43ce4a4c954f4a5f80954b38933c189d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bf81f3d986e428190d3f7727f36d96c in term 2.
I20240501 14:10:48.457464 21070 leader_election.cc:302] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43ce4a4c954f4a5f80954b38933c189d, 5bf81f3d986e428190d3f7727f36d96c; no voters:
I20240501 14:10:48.457641 29806 raft_consensus.cc:2781] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 14:10:48.457686 29806 raft_consensus.cc:482] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c [term 2 FOLLOWER]: Starting leader election (detected failure of leader e6246b183293412494919f1e7efc9448)
I20240501 14:10:48.457722 29806 raft_consensus.cc:3037] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:10:48.458724 29806 raft_consensus.cc:504] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:48.459095 29806 leader_election.cc:288] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c [CANDIDATE]: Term 3 election: Requested vote from peers e6246b183293412494919f1e7efc9448 (127.7.39.134:41911), 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135:38751)
W20240501 14:10:48.459736 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:10:48.459810 21398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "814ba2247a314a4797aa0f9f624aaeb6" candidate_uuid: "5bf81f3d986e428190d3f7727f36d96c" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "43ce4a4c954f4a5f80954b38933c189d"
I20240501 14:10:48.459918 21398 raft_consensus.cc:3037] T 814ba2247a314a4797aa0f9f624aaeb6 P 43ce4a4c954f4a5f80954b38933c189d [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:10:48.460942 21398 raft_consensus.cc:2454] T 814ba2247a314a4797aa0f9f624aaeb6 P 43ce4a4c954f4a5f80954b38933c189d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf81f3d986e428190d3f7727f36d96c in term 3.
I20240501 14:10:48.462630 21070 leader_election.cc:302] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43ce4a4c954f4a5f80954b38933c189d, 5bf81f3d986e428190d3f7727f36d96c; no voters:
I20240501 14:10:48.463750 29806 raft_consensus.cc:2781] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:10:48.463850 29806 raft_consensus.cc:686] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c [term 3 LEADER]: Becoming Leader. State: Replica: 5bf81f3d986e428190d3f7727f36d96c, State: Running, Role: LEADER
I20240501 14:10:48.463989 29806 consensus_queue.cc:237] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
W20240501 14:10:48.464507 21071 leader_election.cc:334] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:10:48.465822 19229 catalog_manager.cc:5484] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c reported cstate change: term changed from 2 to 3, leader changed from e6246b183293412494919f1e7efc9448 (127.7.39.134) to 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133). New cstate: current_term: 3 leader_uuid: "5bf81f3d986e428190d3f7727f36d96c" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
W20240501 14:10:48.467434 21071 leader_election.cc:334] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:10:48.473563 29389 raft_consensus.cc:482] T 12ffa74f7a5b472aac57253693273cad P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e6246b183293412494919f1e7efc9448)
I20240501 14:10:48.473702 29389 raft_consensus.cc:504] T 12ffa74f7a5b472aac57253693273cad P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:48.474145 29389 leader_election.cc:288] T 12ffa74f7a5b472aac57253693273cad P fb1e6bc2d07b4ce3b622a66f14c93bdc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357), e6246b183293412494919f1e7efc9448 (127.7.39.134:41911)
I20240501 14:10:48.475373 19769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12ffa74f7a5b472aac57253693273cad" candidate_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: false dest_uuid: "da3c4e948ff747678f18456c134d7ed4" is_pre_election: true
W20240501 14:10:48.477036 19310 leader_election.cc:334] T 12ffa74f7a5b472aac57253693273cad P fb1e6bc2d07b4ce3b622a66f14c93bdc [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:10:48.477108 19310 leader_election.cc:302] T 12ffa74f7a5b472aac57253693273cad P fb1e6bc2d07b4ce3b622a66f14c93bdc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fb1e6bc2d07b4ce3b622a66f14c93bdc; no voters: da3c4e948ff747678f18456c134d7ed4, e6246b183293412494919f1e7efc9448
I20240501 14:10:48.477520 29389 raft_consensus.cc:2726] T 12ffa74f7a5b472aac57253693273cad P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:10:48.478789 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:48.484283 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:48.484449 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:48.488034 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:48.501340 19748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:48.524276 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:48.534754 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:48.545102 21137 consensus_queue.cc:237] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 5bf81f3d986e428190d3f7727f36d96c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:48.546448 21398 raft_consensus.cc:1261] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 43ce4a4c954f4a5f80954b38933c189d [term 2 FOLLOWER]: Refusing update from remote peer 5bf81f3d986e428190d3f7727f36d96c: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 14:10:48.546855 29806 consensus_queue.cc:1035] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 5bf81f3d986e428190d3f7727f36d96c [LEADER]: Connected to new peer: Peer: permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:10:48.548633 30223 raft_consensus.cc:2932] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 5bf81f3d986e428190d3f7727f36d96c [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:48.549070 21398 raft_consensus.cc:2932] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 43ce4a4c954f4a5f80954b38933c189d [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:48.551554 19506 raft_consensus.cc:1261] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 2 FOLLOWER]: Refusing update from remote peer 5bf81f3d986e428190d3f7727f36d96c: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 14:10:48.552067 29806 consensus_queue.cc:1035] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 5bf81f3d986e428190d3f7727f36d96c [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:10:48.552182 19225 catalog_manager.cc:5484] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 5bf81f3d986e428190d3f7727f36d96c reported cstate change: config changed from index 6 to 8, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) evicted. New cstate: current_term: 2 leader_uuid: "5bf81f3d986e428190d3f7727f36d96c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:48.553053 19506 raft_consensus.cc:2932] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:10:48.555260 19356 tablet_service.cc:1514] Processing DeleteTablet for tablet 98af8b06b9324e3fa7cdb898b7b9fab9 with delete_type TABLET_DATA_TOMBSTONED (TS fb1e6bc2d07b4ce3b622a66f14c93bdc not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:46838
I20240501 14:10:48.558604 30295 tablet_replica.cc:330] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc: stopping tablet replica
I20240501 14:10:48.562084 30295 raft_consensus.cc:2229] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:48.562309 30295 raft_consensus.cc:2258] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:48.563076 30295 ts_tablet_manager.cc:1875] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:48.569670 30295 ts_tablet_manager.cc:1888] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240501 14:10:48.572127 30295 log.cc:1196] T 98af8b06b9324e3fa7cdb898b7b9fab9 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal/wals/98af8b06b9324e3fa7cdb898b7b9fab9
I20240501 14:10:48.573379 19218 catalog_manager.cc:4830] TS fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847): tablet 98af8b06b9324e3fa7cdb898b7b9fab9 (table rebalance_test_table_5 [id=4d7621338643426d90fe03eb71654cb8]) successfully deleted
I20240501 14:10:48.623296 19748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:48.629086 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:48.630717 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:48.633423 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:48.634685 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:48.653187 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:48.663894 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:48.677022 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:48.681768 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:48.687919 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:48.692700 19748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:48.692822 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:48.707209 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:48.710259 29957 raft_consensus.cc:482] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e6246b183293412494919f1e7efc9448)
I20240501 14:10:48.710402 29957 raft_consensus.cc:504] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:48.710887 29957 leader_election.cc:288] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847), e6246b183293412494919f1e7efc9448 (127.7.39.134:41911)
W20240501 14:10:48.712149 19703 leader_election.cc:334] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:10:48.712656 19375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12ffa74f7a5b472aac57253693273cad" candidate_uuid: "da3c4e948ff747678f18456c134d7ed4" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: false dest_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" is_pre_election: true
I20240501 14:10:48.712802 19375 raft_consensus.cc:2454] T 12ffa74f7a5b472aac57253693273cad P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da3c4e948ff747678f18456c134d7ed4 in term 2.
I20240501 14:10:48.713133 19702 leader_election.cc:302] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: da3c4e948ff747678f18456c134d7ed4, fb1e6bc2d07b4ce3b622a66f14c93bdc; no voters: e6246b183293412494919f1e7efc9448
I20240501 14:10:48.713522 29957 raft_consensus.cc:2781] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 14:10:48.713582 29957 raft_consensus.cc:482] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader e6246b183293412494919f1e7efc9448)
I20240501 14:10:48.713621 29957 raft_consensus.cc:3037] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:10:48.714834 29957 raft_consensus.cc:504] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:48.715186 29957 leader_election.cc:288] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 [CANDIDATE]: Term 3 election: Requested vote from peers fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847), e6246b183293412494919f1e7efc9448 (127.7.39.134:41911)
W20240501 14:10:48.716272 19703 leader_election.cc:334] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:10:48.716452 19375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12ffa74f7a5b472aac57253693273cad" candidate_uuid: "da3c4e948ff747678f18456c134d7ed4" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: false dest_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc"
I20240501 14:10:48.716575 19375 raft_consensus.cc:3037] T 12ffa74f7a5b472aac57253693273cad P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:10:48.717557 19375 raft_consensus.cc:2454] T 12ffa74f7a5b472aac57253693273cad P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da3c4e948ff747678f18456c134d7ed4 in term 3.
I20240501 14:10:48.717916 19702 leader_election.cc:302] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: da3c4e948ff747678f18456c134d7ed4, fb1e6bc2d07b4ce3b622a66f14c93bdc; no voters: e6246b183293412494919f1e7efc9448
I20240501 14:10:48.718127 29957 raft_consensus.cc:2781] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:10:48.718253 29957 raft_consensus.cc:686] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 [term 3 LEADER]: Becoming Leader. State: Replica: da3c4e948ff747678f18456c134d7ed4, State: Running, Role: LEADER
I20240501 14:10:48.718394 29957 consensus_queue.cc:237] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:48.720557 19225 catalog_manager.cc:5484] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 reported cstate change: term changed from 2 to 3, leader changed from e6246b183293412494919f1e7efc9448 (127.7.39.134) to da3c4e948ff747678f18456c134d7ed4 (127.7.39.132). New cstate: current_term: 3 leader_uuid: "da3c4e948ff747678f18456c134d7ed4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } } }
I20240501 14:10:48.729884 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:10:48.766064 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:10:48.772935 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:48.780388 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:48.785185 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:48.789681 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:48.802181 21580 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240501 14:10:32.536659 21588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:10:32.536921 21588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:10:32.540868 21588 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:10:33.630404 21588 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 14:10:34.107863 21588 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b52269ef7c494f74a5e4f0d5cb3b8a47: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:34.108974 21588 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3866c35dd0e64d87b6049e5e14cb0d31: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:34.109915 21588 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afc36d383e3e4795ad11804e9bebcb51: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:34.110098 21588 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a9b1c37e923e49a7b30905624977117c: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:34.115895 21588 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 35d6f8fac5394197b987aa44dab206c1: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:34.116273 21588 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1fbbd460d13647c1a85b517926267158: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server e6246b183293412494919f1e7efc9448
W20240501 14:10:34.116345 21588 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4d7621338643426d90fe03eb71654cb8: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:34.116521 21588 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afc36d383e3e4795ad11804e9bebcb51: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server e6246b183293412494919f1e7efc9448
W20240501 14:10:34.117013 21588 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6f6b0a378a484e5889d733ff982d9eb7: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server e6246b183293412494919f1e7efc9448
W20240501 14:10:34.117084 21588 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a9b1c37e923e49a7b30905624977117c: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:34.117136 21588 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 151dfad0ef40470ab9c5203c42400b60: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:34.117189 21588 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ed1a7ed87fa9480ba18a052130ecdc05: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:34.117259 21588 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3866c35dd0e64d87b6049e5e14cb0d31: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:34.117378 21588 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1fbbd460d13647c1a85b517926267158: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:34.117453 21588 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4d7621338643426d90fe03eb71654cb8: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 5bf81f3d986e428190d3f7727f36d96c
I20240501 14:10:34.227541 21588 rebalancer_tool.cc:1074] tablet cf62635e9a994259b9e293e5eb425b55: '99a0be6ef2144b7fb247ba9d3f051d66' -> '778c78105092428bbcde75ab2cdb6a05' move scheduled
I20240501 14:10:34.306392 21588 rebalancer_tool.cc:1074] tablet 7eea58d4a33f422a9ada1e82d408577f: 'ec8cde17a00645e7b38d5d29f7425367' -> '5bf81f3d986e428190d3f7727f36d96c' move scheduled
I20240501 14:10:34.385989 21588 rebalancer_tool.cc:1074] tablet db0f01ee72b24ec9b84410cf94ef9759: 'fb1e6bc2d07b4ce3b622a66f14c93bdc' -> '43ce4a4c954f4a5f80954b38933c189d' move scheduled
I20240501 14:10:34.629591 21588 rebalancer_tool.cc:1074] tablet 814ba2247a314a4797aa0f9f624aaeb6: 'da3c4e948ff747678f18456c134d7ed4' -> 'e6246b183293412494919f1e7efc9448' move scheduled
I20240501 14:10:34.737917 21588 rebalancer_tool.cc:1074] tablet 10aa2feeb0324aa4a6221d2f2f3fc941: '99a0be6ef2144b7fb247ba9d3f051d66' -> '778c78105092428bbcde75ab2cdb6a05' move scheduled
I20240501 14:10:35.076807 21588 rebalancer_tool.cc:1074] tablet 98af8b06b9324e3fa7cdb898b7b9fab9: 'ec8cde17a00645e7b38d5d29f7425367' -> '5bf81f3d986e428190d3f7727f36d96c' move scheduled
I20240501 14:10:35.147133 21588 rebalancer_tool.cc:1084] tablet 85593473500b4d5eaf9f8b0bf2296777: 'fb1e6bc2d07b4ce3b622a66f14c93bdc' -> 'e6246b183293412494919f1e7efc9448' move ignored: Illegal state: tablet 85593473500b4d5eaf9f8b0bf2296777: replica e6246b183293412494919f1e7efc9448 is already present
I20240501 14:10:35.147228 21588 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:10:35.506517 21588 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a9b1c37e923e49a7b30905624977117c: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:35.506695 21588 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 35d6f8fac5394197b987aa44dab206c1: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:35.506814 21588 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3866c35dd0e64d87b6049e5e14cb0d31: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:35.506906 21588 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ed1a7ed87fa9480ba18a052130ecdc05: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:35.507026 21588 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6f6b0a378a484e5889d733ff982d9eb7: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:35.507160 21588 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afc36d383e3e4795ad11804e9bebcb51: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:35.507225 21588 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 151dfad0ef40470ab9c5203c42400b60: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server e6246b183293412494919f1e7efc9448
W20240501 14:10:35.507285 21588 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1fbbd460d13647c1a85b517926267158: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:35.507365 21588 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b52269ef7c494f74a5e4f0d5cb3b8a47: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:35.507423 21588 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4d7621338643426d90fe03eb71654cb8: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:35.507473 21588 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3866c35dd0e64d87b6049e5e14cb0d31: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:35.507551 21588 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ed1a7ed87fa9480ba18a052130ecdc05: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:35.507992 21588 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a9b1c37e923e49a7b30905624977117c: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:35.508116 21588 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6f6b0a378a484e5889d733ff982d9eb7: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server e6246b183293412494919f1e7efc9448
I20240501 14:10:35.530526 21588 rebalancer_tool.cc:1074] tablet 85593473500b4d5eaf9f8b0bf2296777: 'fb1e6bc2d07b4ce3b622a66f14c93bdc' -> '' move scheduled
I20240501 14:10:35.623308 21588 rebalancer_tool.cc:1074] tablet fe554c3145de46cf8070efcecffbaedb: 'da3c4e948ff747678f18456c134d7ed4' -> 'e6246b183293412494919f1e7efc9448' move scheduled
I20240501 14:10:35.784345 21588 rebalancer_tool.cc:1074] tablet cc8e888d46d34a22b6d4409ec2ab53a9: '99a0be6ef2144b7fb247ba9d3f051d66' -> '43ce4a4c954f4a5f80954b38933c189d' move scheduled
I20240501 14:10:35.800431 21588 rebalancer_tool.cc:1074] tablet b6be9a115fe14c098c29cad819849f95: 'ec8cde17a00645e7b38d5d29f7425367' -> '' move scheduled
I20240501 14:10:35.811656 21588 rebalancer_tool.cc:1084] tablet 1d408b075d784953a57554bcdf19158a: 'fb1e6bc2d07b4ce3b622a66f14c93bdc' -> '5bf81f3d986e428190d3f7727f36d96c' move ignored: Illegal state: tablet 1d408b075d784953a57554bcdf19158a: replica 5bf81f3d986e428190d3f7727f36d96c is already present
I20240501 14:10:35.811729 21588 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:10:36.390635 21588 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ed1a7ed87fa9480ba18a052130ecdc05: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:36.390920 21588 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 35d6f8fac5394197b987aa44dab206c1: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server e6246b183293412494919f1e7efc9448
W20240501 14:10:36.390972 21588 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afc36d383e3e4795ad11804e9bebcb51: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:36.391002 21588 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3866c35dd0e64d87b6049e5e14cb0d31: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:36.391033 21588 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a9b1c37e923e49a7b30905624977117c: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:36.391124 21588 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a9b1c37e923e49a7b30905624977117c: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:36.391157 21588 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1fbbd460d13647c1a85b517926267158: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:36.391201 21588 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6f6b0a378a484e5889d733ff982d9eb7: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server e6246b183293412494919f1e7efc9448
W20240501 14:10:36.391242 21588 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4d7621338643426d90fe03eb71654cb8: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:36.391281 21588 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b52269ef7c494f74a5e4f0d5cb3b8a47: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:36.391324 21588 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 151dfad0ef40470ab9c5203c42400b60: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server e6246b183293412494919f1e7efc9448
W20240501 14:10:36.391408 21588 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3866c35dd0e64d87b6049e5e14cb0d31: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:36.391443 21588 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b52269ef7c494f74a5e4f0d5cb3b8a47: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server 5bf81f3d986e428190d3f7727f36d96c
I20240501 14:10:36.551610 21588 rebalancer_tool.cc:1074] tablet 78e14dea8b6e4c0abd1c88dcd5b4e608: 'da3c4e948ff747678f18456c134d7ed4' -> '5bf81f3d986e428190d3f7727f36d96c' move scheduled
I20240501 14:10:36.577775 21588 rebalancer_tool.cc:1084] tablet 12ffa74f7a5b472aac57253693273cad: 'fb1e6bc2d07b4ce3b622a66f14c93bdc' -> 'e6246b183293412494919f1e7efc9448' move ignored: Illegal state: tablet 12ffa74f7a5b472aac57253693273cad: replica e6246b183293412494919f1e7efc9448 is already present
I20240501 14:10:36.577884 21588 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:10:37.242537 21588 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a9b1c37e923e49a7b30905624977117c: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:37.242738 21588 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b52269ef7c494f74a5e4f0d5cb3b8a47: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:37.242807 21588 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a9b1c37e923e49a7b30905624977117c: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server e6246b183293412494919f1e7efc9448
W20240501 14:10:37.242894 21588 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1fbbd460d13647c1a85b517926267158: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server e6246b183293412494919f1e7efc9448
W20240501 14:10:37.242944 21588 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 151dfad0ef40470ab9c5203c42400b60: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:37.242986 21588 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 35d6f8fac5394197b987aa44dab206c1: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:37.243083 21588 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4d7621338643426d90fe03eb71654cb8: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 778c78105092428bbcde75ab2cdb6a05
I20240501 14:10:37.262476 21588 rebalancer_tool.cc:1074] tablet f6ca088ce3aa4b97aa54c8ce6d700921: 'fb1e6bc2d07b4ce3b622a66f14c93bdc' -> '' move scheduled
I20240501 14:10:37.270653 21588 rebalancer_tool.cc:1074] tablet d63afd081986473aa2715d074b123a93: '99a0be6ef2144b7fb247ba9d3f051d66' -> '' move scheduled
I20240501 14:10:37.286944 21588 rebalancer_tool.cc:1074] tablet f69020bb6c244848a6b0a50fd2b4f3eb: 'ec8cde17a00645e7b38d5d29f7425367' -> '' move scheduled
I20240501 14:10:39.706161 21588 rebalancer_tool.cc:1136] tablet 7eea58d4a33f422a9ada1e82d408577f: 'ec8cde17a00645e7b38d5d29f7425367' -> '5bf81f3d986e428190d3f7727f36d96c' move completed: OK
I20240501 14:10:42.037412 21588 rebalancer_tool.cc:1136] tablet 85593473500b4d5eaf9f8b0bf2296777: 'fb1e6bc2d07b4ce3b622a66f14c93bdc' -> '' move completed: OK
I20240501 14:10:42.048415 21588 rebalancer_tool.cc:1136] tablet cf62635e9a994259b9e293e5eb425b55: '99a0be6ef2144b7fb247ba9d3f051d66' -> '778c78105092428bbcde75ab2cdb6a05' move completed: OK
I20240501 14:10:43.561837 21588 rebalancer_tool.cc:1136] tablet d63afd081986473aa2715d074b123a93: '99a0be6ef2144b7fb247ba9d3f051d66' -> '' move completed: OK
I20240501 14:10:44.213052 21588 rebalancer_tool.cc:1074] tablet bb291a0c743c4e638f79da36cc77faa3: 'fb1e6bc2d07b4ce3b622a66f14c93bdc' -> '' move scheduled
I20240501 14:10:44.295933 21588 rebalancer_tool.cc:1074] tablet 410a7dd0dd734684a82875912966657d: '99a0be6ef2144b7fb247ba9d3f051d66' -> '' move scheduled
I20240501 14:10:44.917069 21588 rebalancer_tool.cc:1136] tablet f69020bb6c244848a6b0a50fd2b4f3eb: 'ec8cde17a00645e7b38d5d29f7425367' -> '' move completed: OK
I20240501 14:10:44.928543 21588 rebalancer_tool.cc:1136] tablet b6be9a115fe14c098c29cad819849f95: 'ec8cde17a00645e7b38d5d29f7425367' -> '' move completed: OK
I20240501 14:10:44.955718 21588 rebalancer_tool.cc:1136] tablet 78e14dea8b6e4c0abd1c88dcd5b4e608: 'da3c4e948ff747678f18456c134d7ed4' -> '5bf81f3d986e428190d3f7727f36d96c' move completed: OK
I20240501 14:10:44.969669 21588 rebalancer_tool.cc:1136] tablet bb291a0c743c4e638f79da36cc77faa3: 'fb1e6bc2d07b4ce3b622a66f14c93bdc' -> '' move completed: OK
I20240501 14:10:45.021548 21588 rebalancer_tool.cc:1136] tablet f6ca088ce3aa4b97aa54c8ce6d700921: 'fb1e6bc2d07b4ce3b622a66f14c93bdc' -> '' move completed: OK
I20240501 14:10:45.032682 21588 rebalancer_tool.cc:1136] tablet fe554c3145de46cf8070efcecffbaedb: 'da3c4e948ff747678f18456c134d7ed4' -> 'e6246b183293412494919f1e7efc9448' move completed: OK
I20240501 14:10:45.139236 21588 rebalancer_tool.cc:1136] tablet 814ba2247a314a4797aa0f9f624aaeb6: 'da3c4e948ff747678f18456c134d7ed4' -> 'e6246b183293412494919f1e7efc9448' move completed: OK
I20240501 14:10:45.157192 21588 rebalancer_tool.cc:1136] tablet 10aa2feeb0324aa4a6221d2f2f3fc941: '99a0be6ef2144b7fb247ba9d3f051d66' -> '778c78105092428bbcde75ab2cdb6a05' move completed: OK
I20240501 14:10:45.207813 21588 rebalancer_tool.cc:1136] tablet cc8e888d46d34a22b6d4409ec2ab53a9: '99a0be6ef2144b7fb247ba9d3f051d66' -> '43ce4a4c954f4a5f80954b38933c189d' move completed: OK
I20240501 14:10:45.298813 21588 rebalancer_tool.cc:1136] tablet db0f01ee72b24ec9b84410cf94ef9759: 'fb1e6bc2d07b4ce3b622a66f14c93bdc' -> '43ce4a4c954f4a5f80954b38933c189d' move completed: OK
I20240501 14:10:45.356757 21588 rebalancer_tool.cc:1136] tablet 98af8b06b9324e3fa7cdb898b7b9fab9: 'ec8cde17a00645e7b38d5d29f7425367' -> '5bf81f3d986e428190d3f7727f36d96c' move completed: OK
I20240501 14:10:45.369709 21588 rebalancer_tool.cc:1074] tablet ad7c255baee74e629e2976a942a4174f: 'da3c4e948ff747678f18456c134d7ed4' -> '' move scheduled
I20240501 14:10:45.423403 21588 rebalancer_tool.cc:1074] tablet 11f64b59f83e458f8cf51fd5bcd36fee: 'fb1e6bc2d07b4ce3b622a66f14c93bdc' -> '' move scheduled
I20240501 14:10:45.437983 21588 rebalancer_tool.cc:1074] tablet ec2a46f8cfb3406082970e5fdb59b101: 'ec8cde17a00645e7b38d5d29f7425367' -> '' move scheduled
I20240501 14:10:45.466671 21588 rebalancer_tool.cc:1074] tablet 2dcca6e0e6374520bd4522c80c110e97: '99a0be6ef2144b7fb247ba9d3f051d66' -> '' move scheduled
I20240501 14:10:45.477433 21588 rebalancer_tool.cc:1136] tablet ec2a46f8cfb3406082970e5fdb59b101: 'ec8cde17a00645e7b38d5d29f7425367' -> '' move completed: OK
I20240501 14:10:45.490533 21588 rebalancer_tool.cc:1136] tablet 11f64b59f83e458f8cf51fd5bcd36fee: 'fb1e6bc2d07b4ce3b622a66f14c93bdc' -> '' move completed: OK
I20240501 14:10:45.599115 21588 rebalancer_tool.cc:1136] tablet 2dcca6e0e6374520bd4522c80c110e97: '99a0be6ef2144b7fb247ba9d3f051d66' -> '' move completed: OK
I20240501 14:10:45.752022 21588 rebalancer_tool.cc:1136] tablet ad7c255baee74e629e2976a942a4174f: 'da3c4e948ff747678f18456c134d7ed4' -> '' move completed: OK
I20240501 14:10:45.781558 21588 rebalancer_tool.cc:1136] tablet 410a7dd0dd734684a82875912966657d: '99a0be6ef2144b7fb247ba9d3f051d66' -> '' move completed: OK
I20240501 14:10:45.804656 21588 rebalancer_tool.cc:1074] tablet 0988c468757d49738765bdcb31794bd1: 'ec8cde17a00645e7b38d5d29f7425367' -> '' move scheduled
I20240501 14:10:45.837941 21588 rebalancer_tool.cc:1074] tablet 1d408b075d784953a57554bcdf19158a: 'fb1e6bc2d07b4ce3b622a66f14c93bdc' -> '' move scheduled
I20240501 14:10:45.848649 21588 rebalancer_tool.cc:1074] tablet 272eee3450cf4f48a4e1de2dcb28c0f7: 'ec8cde17a00645e7b38d5d29f7425367' -> '' move scheduled
I20240501 14:10:46.509022 21588 rebalancer_tool.cc:1136] tablet 1d408b075d784953a57554bcdf19158a: 'fb1e6bc2d07b4ce3b622a66f14c93bdc' -> '' move completed: OK
I20240501 14:10:46.585705 21588 rebalancer_tool.cc:1136] tablet 0988c468757d49738765bdcb31794bd1: 'ec8cde17a00645e7b38d5d29f7425367' -> '' move completed: OK
I20240501 14:10:46.600705 21588 rebalancer_tool.cc:1136] tablet 272eee3450cf4f48a4e1de2dcb28c0f7: 'ec8cde17a00645e7b38d5d29f7425367' -> '' move completed: OK
W20240501 14:10:47.300503 21588 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b52269ef7c494f74a5e4f0d5cb3b8a47: could not find any suitable replica to move from server 5bf81f3d986e428190d3f7727f36d96c to server 99a0be6ef2144b7fb247ba9d3f051d66
I20240501 14:10:48.451090 21588 rebalancer_tool.cc:1074] tablet db0f01ee72b24ec9b84410cf94ef9759: 'e6246b183293412494919f1e7efc9448' -> 'da3c4e948ff747678f18456c134d7ed4' move scheduled
I20240501 14:10:48.470012 21588 rebalancer_tool.cc:1084] tablet 814ba2247a314a4797aa0f9f624aaeb6: '43ce4a4c954f4a5f80954b38933c189d' -> '99a0be6ef2144b7fb247ba9d3f051d66' move ignored: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:10:48.470099 21588 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): unacceptable health status UNAVAILABLE
I20240501 14:10:48.820998 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:48.826396 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:48.827208 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:48.827545 30384 ts_tablet_manager.cc:902] T cf62635e9a994259b9e293e5eb425b55 P 5bf81f3d986e428190d3f7727f36d96c: Initiating tablet copy from peer ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201)
I20240501 14:10:48.828092 30384 tablet_copy_client.cc:308] T cf62635e9a994259b9e293e5eb425b55 P 5bf81f3d986e428190d3f7727f36d96c: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.131:45201
I20240501 14:10:48.833027 19657 tablet_copy_service.cc:140] P ec8cde17a00645e7b38d5d29f7425367: Received BeginTabletCopySession request for tablet cf62635e9a994259b9e293e5eb425b55 from peer 5bf81f3d986e428190d3f7727f36d96c ({username='slave'} at 127.7.39.133:54557)
I20240501 14:10:48.833144 19657 tablet_copy_service.cc:161] P ec8cde17a00645e7b38d5d29f7425367: Beginning new tablet copy session on tablet cf62635e9a994259b9e293e5eb425b55 from peer 5bf81f3d986e428190d3f7727f36d96c at {username='slave'} at 127.7.39.133:54557: session id = 5bf81f3d986e428190d3f7727f36d96c-cf62635e9a994259b9e293e5eb425b55
I20240501 14:10:48.834115 19657 tablet_copy_source_session.cc:205] T cf62635e9a994259b9e293e5eb425b55 P ec8cde17a00645e7b38d5d29f7425367: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:48.834980 30384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf62635e9a994259b9e293e5eb425b55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:48.837497 30384 tablet_copy_client.cc:783] T cf62635e9a994259b9e293e5eb425b55 P 5bf81f3d986e428190d3f7727f36d96c: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:48.837836 30384 tablet_copy_client.cc:647] T cf62635e9a994259b9e293e5eb425b55 P 5bf81f3d986e428190d3f7727f36d96c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:48.856201 30384 tablet_copy_client.cc:523] T cf62635e9a994259b9e293e5eb425b55 P 5bf81f3d986e428190d3f7727f36d96c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:48.858729 30384 tablet_bootstrap.cc:492] T cf62635e9a994259b9e293e5eb425b55 P 5bf81f3d986e428190d3f7727f36d96c: Bootstrap starting.
I20240501 14:10:48.864598 21522 raft_consensus.cc:1203] T 1d408b075d784953a57554bcdf19158a P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240501 14:10:48.894584 30384 tablet_bootstrap.cc:492] T cf62635e9a994259b9e293e5eb425b55 P 5bf81f3d986e428190d3f7727f36d96c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:48.894995 30384 tablet_bootstrap.cc:492] T cf62635e9a994259b9e293e5eb425b55 P 5bf81f3d986e428190d3f7727f36d96c: Bootstrap complete.
I20240501 14:10:48.895336 30384 ts_tablet_manager.cc:1367] T cf62635e9a994259b9e293e5eb425b55 P 5bf81f3d986e428190d3f7727f36d96c: Time spent bootstrapping tablet: real 0.037s user 0.028s sys 0.000s
I20240501 14:10:48.896188 30384 raft_consensus.cc:348] T cf62635e9a994259b9e293e5eb425b55 P 5bf81f3d986e428190d3f7727f36d96c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:48.896330 30384 raft_consensus.cc:729] T cf62635e9a994259b9e293e5eb425b55 P 5bf81f3d986e428190d3f7727f36d96c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf81f3d986e428190d3f7727f36d96c, State: Initialized, Role: FOLLOWER
I20240501 14:10:48.896548 30384 consensus_queue.cc:260] T cf62635e9a994259b9e293e5eb425b55 P 5bf81f3d986e428190d3f7727f36d96c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:10:48.898907 30384 ts_tablet_manager.cc:1398] T cf62635e9a994259b9e293e5eb425b55 P 5bf81f3d986e428190d3f7727f36d96c: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:10:48.899780 19657 tablet_copy_service.cc:342] P ec8cde17a00645e7b38d5d29f7425367: Request end of tablet copy session 5bf81f3d986e428190d3f7727f36d96c-cf62635e9a994259b9e293e5eb425b55 received from {username='slave'} at 127.7.39.133:54557
I20240501 14:10:48.899870 19657 tablet_copy_service.cc:433] P ec8cde17a00645e7b38d5d29f7425367: ending tablet copy session 5bf81f3d986e428190d3f7727f36d96c-cf62635e9a994259b9e293e5eb425b55 on tablet cf62635e9a994259b9e293e5eb425b55 with peer 5bf81f3d986e428190d3f7727f36d96c
I20240501 14:10:48.914403 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:48.916393 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:48.927635 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:48.927687 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:48.928547 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:48.930280 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:48.932758 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:49.019850 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:49.025035 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:49.026535 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:49.034914 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:49.039016 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:49.041640 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:49.042094 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:49.064945 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:49.070489 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:49.073940 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:49.080770 21398 raft_consensus.cc:1261] T 814ba2247a314a4797aa0f9f624aaeb6 P 43ce4a4c954f4a5f80954b38933c189d [term 3 FOLLOWER]: Refusing update from remote peer 5bf81f3d986e428190d3f7727f36d96c: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240501 14:10:49.081149 29806 consensus_queue.cc:1035] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c [LEADER]: Connected to new peer: Peer: permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:10:49.089922 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:49.090457 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:49.096055 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:10:49.105319 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:49.105870 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:49.107834 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:49.148780 21579 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240501 14:10:32.583617 21586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:10:32.583933 21586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:10:32.588133 21586 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:10:33.648030 21586 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 14:10:34.160636 21586 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a9b1c37e923e49a7b30905624977117c: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:34.160773 21586 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b52269ef7c494f74a5e4f0d5cb3b8a47: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:34.161053 21586 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 35d6f8fac5394197b987aa44dab206c1: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server e6246b183293412494919f1e7efc9448
W20240501 14:10:34.161109 21586 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afc36d383e3e4795ad11804e9bebcb51: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server e6246b183293412494919f1e7efc9448
W20240501 14:10:34.161136 21586 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a9b1c37e923e49a7b30905624977117c: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:34.161170 21586 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4d7621338643426d90fe03eb71654cb8: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server e6246b183293412494919f1e7efc9448
W20240501 14:10:34.161198 21586 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1fbbd460d13647c1a85b517926267158: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:34.161224 21586 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6f6b0a378a484e5889d733ff982d9eb7: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:34.161252 21586 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b52269ef7c494f74a5e4f0d5cb3b8a47: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:34.161276 21586 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ed1a7ed87fa9480ba18a052130ecdc05: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:34.161302 21586 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 151dfad0ef40470ab9c5203c42400b60: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:34.161341 21586 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3866c35dd0e64d87b6049e5e14cb0d31: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:34.161396 21586 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afc36d383e3e4795ad11804e9bebcb51: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server e6246b183293412494919f1e7efc9448
W20240501 14:10:34.161434 21586 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a9b1c37e923e49a7b30905624977117c: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:34.161479 21586 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4d7621338643426d90fe03eb71654cb8: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:34.161516 21586 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6f6b0a378a484e5889d733ff982d9eb7: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 778c78105092428bbcde75ab2cdb6a05
I20240501 14:10:34.269243 21586 rebalancer_tool.cc:1074] tablet 85593473500b4d5eaf9f8b0bf2296777: 'fb1e6bc2d07b4ce3b622a66f14c93bdc' -> 'e6246b183293412494919f1e7efc9448' move scheduled
I20240501 14:10:34.353826 21586 rebalancer_tool.cc:1074] tablet ad7c255baee74e629e2976a942a4174f: '99a0be6ef2144b7fb247ba9d3f051d66' -> '43ce4a4c954f4a5f80954b38933c189d' move scheduled
I20240501 14:10:34.496694 21586 rebalancer_tool.cc:1074] tablet 0988c468757d49738765bdcb31794bd1: 'ec8cde17a00645e7b38d5d29f7425367' -> '5bf81f3d986e428190d3f7727f36d96c' move scheduled
I20240501 14:10:34.622640 21586 rebalancer_tool.cc:1074] tablet a9edf1764b55420ea5d519e07ae1d5e0: 'da3c4e948ff747678f18456c134d7ed4' -> '778c78105092428bbcde75ab2cdb6a05' move scheduled
I20240501 14:10:34.874223 21586 rebalancer_tool.cc:1074] tablet 11f64b59f83e458f8cf51fd5bcd36fee: 'fb1e6bc2d07b4ce3b622a66f14c93bdc' -> 'e6246b183293412494919f1e7efc9448' move scheduled
I20240501 14:10:35.010253 21586 rebalancer_tool.cc:1074] tablet 10aa2feeb0324aa4a6221d2f2f3fc941: '99a0be6ef2144b7fb247ba9d3f051d66' -> '43ce4a4c954f4a5f80954b38933c189d' move scheduled
I20240501 14:10:35.124589 21586 rebalancer_tool.cc:1074] tablet ec2a46f8cfb3406082970e5fdb59b101: 'ec8cde17a00645e7b38d5d29f7425367' -> '5bf81f3d986e428190d3f7727f36d96c' move scheduled
I20240501 14:10:35.369604 21586 rebalancer_tool.cc:1074] tablet 272eee3450cf4f48a4e1de2dcb28c0f7: 'da3c4e948ff747678f18456c134d7ed4' -> '778c78105092428bbcde75ab2cdb6a05' move scheduled
I20240501 14:10:35.571776 21586 rebalancer_tool.cc:1074] tablet cc8e888d46d34a22b6d4409ec2ab53a9: 'fb1e6bc2d07b4ce3b622a66f14c93bdc' -> 'e6246b183293412494919f1e7efc9448' move scheduled
I20240501 14:10:35.656670 21586 rebalancer_tool.cc:1074] tablet 2dcca6e0e6374520bd4522c80c110e97: '99a0be6ef2144b7fb247ba9d3f051d66' -> '43ce4a4c954f4a5f80954b38933c189d' move scheduled
I20240501 14:10:35.808964 21586 rebalancer_tool.cc:1074] tablet bb291a0c743c4e638f79da36cc77faa3: 'ec8cde17a00645e7b38d5d29f7425367' -> '778c78105092428bbcde75ab2cdb6a05' move scheduled
I20240501 14:10:35.892386 21586 rebalancer_tool.cc:1084] tablet f6ca088ce3aa4b97aa54c8ce6d700921: 'da3c4e948ff747678f18456c134d7ed4' -> '5bf81f3d986e428190d3f7727f36d96c' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240501 14:10:35.892508 21586 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:10:36.334138 21586 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afc36d383e3e4795ad11804e9bebcb51: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:36.334239 21586 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3866c35dd0e64d87b6049e5e14cb0d31: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:36.334337 21586 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a9b1c37e923e49a7b30905624977117c: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:36.334383 21586 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3866c35dd0e64d87b6049e5e14cb0d31: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:36.334427 21586 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a9b1c37e923e49a7b30905624977117c: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:36.334467 21586 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ed1a7ed87fa9480ba18a052130ecdc05: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server e6246b183293412494919f1e7efc9448
W20240501 14:10:36.334538 21586 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1fbbd460d13647c1a85b517926267158: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:36.334581 21586 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afc36d383e3e4795ad11804e9bebcb51: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:36.334637 21586 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b52269ef7c494f74a5e4f0d5cb3b8a47: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:36.334702 21586 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a9b1c37e923e49a7b30905624977117c: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 778c78105092428bbcde75ab2cdb6a05
I20240501 14:10:36.434860 21586 rebalancer_tool.cc:1074] tablet 410a7dd0dd734684a82875912966657d: 'da3c4e948ff747678f18456c134d7ed4' -> '' move scheduled
I20240501 14:10:36.465692 21586 rebalancer_tool.cc:1074] tablet 7eea58d4a33f422a9ada1e82d408577f: 'ec8cde17a00645e7b38d5d29f7425367' -> '' move scheduled
I20240501 14:10:36.567137 21586 rebalancer_tool.cc:1084] tablet dadab886b03540579d138cdb5b25e286: '99a0be6ef2144b7fb247ba9d3f051d66' -> '5bf81f3d986e428190d3f7727f36d96c' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240501 14:10:36.567217 21586 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:10:36.767040 23884 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTabletLocations -> {remote=127.7.39.190:34571, user_credentials={real_user=slave}} blocked reactor thread for 63600.6us
W20240501 14:10:37.150928 21586 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a9b1c37e923e49a7b30905624977117c: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:37.151082 21586 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3866c35dd0e64d87b6049e5e14cb0d31: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:37.151248 21586 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a9b1c37e923e49a7b30905624977117c: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:37.151314 21586 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1fbbd460d13647c1a85b517926267158: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:37.151364 21586 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ed1a7ed87fa9480ba18a052130ecdc05: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:37.151448 21586 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afc36d383e3e4795ad11804e9bebcb51: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:37.151497 21586 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6f6b0a378a484e5889d733ff982d9eb7: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:37.151554 21586 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a9b1c37e923e49a7b30905624977117c: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server e6246b183293412494919f1e7efc9448
I20240501 14:10:37.211477 21586 rebalancer_tool.cc:1074] tablet f69020bb6c244848a6b0a50fd2b4f3eb: '99a0be6ef2144b7fb247ba9d3f051d66' -> '' move scheduled
I20240501 14:10:37.226111 21586 rebalancer_tool.cc:1074] tablet 55830424a98348e382759c54afdeabcd: 'fb1e6bc2d07b4ce3b622a66f14c93bdc' -> '' move scheduled
I20240501 14:10:37.234457 21586 rebalancer_tool.cc:1074] tablet d63afd081986473aa2715d074b123a93: 'da3c4e948ff747678f18456c134d7ed4' -> '' move scheduled
I20240501 14:10:37.294221 21586 rebalancer_tool.cc:1074] tablet ffb4b6a2c19b497f8117fbbc9ea50082: '99a0be6ef2144b7fb247ba9d3f051d66' -> 'e6246b183293412494919f1e7efc9448' move scheduled
I20240501 14:10:41.120874 21586 rebalancer_tool.cc:1136] tablet ad7c255baee74e629e2976a942a4174f: '99a0be6ef2144b7fb247ba9d3f051d66' -> '43ce4a4c954f4a5f80954b38933c189d' move completed: OK
I20240501 14:10:42.323150 21586 rebalancer_tool.cc:1123] tablet 10aa2feeb0324aa4a6221d2f2f3fc941: '99a0be6ef2144b7fb247ba9d3f051d66' -> '43ce4a4c954f4a5f80954b38933c189d' move is abandoned: Not found: Server with UUID 99a0be6ef2144b7fb247ba9d3f051d66 not a member of the config. RaftConfig: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:10:42.451175 21586 rebalancer_tool.cc:1136] tablet ec2a46f8cfb3406082970e5fdb59b101: 'ec8cde17a00645e7b38d5d29f7425367' -> '5bf81f3d986e428190d3f7727f36d96c' move completed: OK
I20240501 14:10:43.622495 21586 rebalancer_tool.cc:1136] tablet 55830424a98348e382759c54afdeabcd: 'fb1e6bc2d07b4ce3b622a66f14c93bdc' -> '' move completed: OK
I20240501 14:10:43.637411 21586 rebalancer_tool.cc:1136] tablet cc8e888d46d34a22b6d4409ec2ab53a9: 'fb1e6bc2d07b4ce3b622a66f14c93bdc' -> 'e6246b183293412494919f1e7efc9448' move completed: OK
I20240501 14:10:43.690937 21586 rebalancer_tool.cc:1136] tablet 7eea58d4a33f422a9ada1e82d408577f: 'ec8cde17a00645e7b38d5d29f7425367' -> '' move completed: OK
I20240501 14:10:43.799264 21586 rebalancer_tool.cc:1136] tablet a9edf1764b55420ea5d519e07ae1d5e0: 'da3c4e948ff747678f18456c134d7ed4' -> '778c78105092428bbcde75ab2cdb6a05' move completed: OK
I20240501 14:10:44.198668 21586 rebalancer_tool.cc:1136] tablet d63afd081986473aa2715d074b123a93: 'da3c4e948ff747678f18456c134d7ed4' -> '' move completed: OK
I20240501 14:10:44.206066 21586 rebalancer_tool.cc:1074] tablet db0f01ee72b24ec9b84410cf94ef9759: 'fb1e6bc2d07b4ce3b622a66f14c93bdc' -> '' move scheduled
I20240501 14:10:44.230021 21586 rebalancer_tool.cc:1074] tablet b6be9a115fe14c098c29cad819849f95: 'ec8cde17a00645e7b38d5d29f7425367' -> '' move scheduled
I20240501 14:10:44.318780 21586 rebalancer_tool.cc:1074] tablet f6ca088ce3aa4b97aa54c8ce6d700921: 'da3c4e948ff747678f18456c134d7ed4' -> '' move scheduled
I20240501 14:10:44.370682 21586 rebalancer_tool.cc:1136] tablet b6be9a115fe14c098c29cad819849f95: 'ec8cde17a00645e7b38d5d29f7425367' -> '' move completed: OK
I20240501 14:10:44.445160 21586 rebalancer_tool.cc:1136] tablet db0f01ee72b24ec9b84410cf94ef9759: 'fb1e6bc2d07b4ce3b622a66f14c93bdc' -> '' move completed: OK
I20240501 14:10:44.520810 21586 rebalancer_tool.cc:1136] tablet 11f64b59f83e458f8cf51fd5bcd36fee: 'fb1e6bc2d07b4ce3b622a66f14c93bdc' -> 'e6246b183293412494919f1e7efc9448' move completed: OK
I20240501 14:10:44.544878 21586 rebalancer_tool.cc:1136] tablet f69020bb6c244848a6b0a50fd2b4f3eb: '99a0be6ef2144b7fb247ba9d3f051d66' -> '' move completed: OK
I20240501 14:10:44.552629 21586 rebalancer_tool.cc:1136] tablet 0988c468757d49738765bdcb31794bd1: 'ec8cde17a00645e7b38d5d29f7425367' -> '5bf81f3d986e428190d3f7727f36d96c' move completed: OK
I20240501 14:10:44.564172 21586 rebalancer_tool.cc:1136] tablet bb291a0c743c4e638f79da36cc77faa3: 'ec8cde17a00645e7b38d5d29f7425367' -> '778c78105092428bbcde75ab2cdb6a05' move completed: OK
I20240501 14:10:44.573508 21586 rebalancer_tool.cc:1136] tablet 85593473500b4d5eaf9f8b0bf2296777: 'fb1e6bc2d07b4ce3b622a66f14c93bdc' -> 'e6246b183293412494919f1e7efc9448' move completed: OK
I20240501 14:10:44.599246 21586 rebalancer_tool.cc:1136] tablet 2dcca6e0e6374520bd4522c80c110e97: '99a0be6ef2144b7fb247ba9d3f051d66' -> '43ce4a4c954f4a5f80954b38933c189d' move completed: OK
I20240501 14:10:44.681046 21586 rebalancer_tool.cc:1136] tablet ffb4b6a2c19b497f8117fbbc9ea50082: '99a0be6ef2144b7fb247ba9d3f051d66' -> 'e6246b183293412494919f1e7efc9448' move completed: OK
I20240501 14:10:44.736035 21586 rebalancer_tool.cc:1136] tablet 272eee3450cf4f48a4e1de2dcb28c0f7: 'da3c4e948ff747678f18456c134d7ed4' -> '778c78105092428bbcde75ab2cdb6a05' move completed: OK
I20240501 14:10:44.744184 21586 rebalancer_tool.cc:1136] tablet f6ca088ce3aa4b97aa54c8ce6d700921: 'da3c4e948ff747678f18456c134d7ed4' -> '' move completed: OK
I20240501 14:10:44.793352 21586 rebalancer_tool.cc:1136] tablet 410a7dd0dd734684a82875912966657d: 'da3c4e948ff747678f18456c134d7ed4' -> '' move completed: OK
I20240501 14:10:44.809937 21586 rebalancer_tool.cc:1074] tablet 98af8b06b9324e3fa7cdb898b7b9fab9: 'fb1e6bc2d07b4ce3b622a66f14c93bdc' -> '' move scheduled
I20240501 14:10:44.826370 21586 rebalancer_tool.cc:1074] tablet cf62635e9a994259b9e293e5eb425b55: 'da3c4e948ff747678f18456c134d7ed4' -> '' move scheduled
I20240501 14:10:44.867882 21586 rebalancer_tool.cc:1074] tablet 1d408b075d784953a57554bcdf19158a: '99a0be6ef2144b7fb247ba9d3f051d66' -> '' move scheduled
W20240501 14:10:47.769018 29838 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:10:47.741948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.39.134:41911 (local address 127.0.0.1:54804)
0501 14:10:47.745565 (+ 3617us) negotiation.cc:107] Waiting for socket to connect
0501 14:10:47.745577 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 14:10:47.745648 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:10:47.768448 (+ 22800us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":3557}
I20240501 14:10:48.807919 21586 rebalancer_tool.cc:1136] tablet 1d408b075d784953a57554bcdf19158a: '99a0be6ef2144b7fb247ba9d3f051d66' -> '' move completed: OK
I20240501 14:10:48.847872 21586 rebalancer_tool.cc:1136] tablet cf62635e9a994259b9e293e5eb425b55: 'da3c4e948ff747678f18456c134d7ed4' -> '' move completed: OK
I20240501 14:10:48.860766 21586 rebalancer_tool.cc:1136] tablet 98af8b06b9324e3fa7cdb898b7b9fab9: 'fb1e6bc2d07b4ce3b622a66f14c93bdc' -> '' move completed: OK
Illegal state: tablet server e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): unacceptable health status UNAVAILABLE
I20240501 14:10:49.199855 19375 raft_consensus.cc:1261] T 12ffa74f7a5b472aac57253693273cad P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 3 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20240501 14:10:49.202642 30351 consensus_queue.cc:1035] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:10:49.209491 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:49.212888 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:49.215253 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:49.224185 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:49.224912 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:49.239166 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:49.240286 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:49.296703 21136 raft_consensus.cc:1203] T cf62635e9a994259b9e293e5eb425b55 P 5bf81f3d986e428190d3f7727f36d96c [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240501 14:10:49.334018 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:49.336800 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:10:49.342353 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:49.343695 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:49.348443 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:49.361495 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:49.367112 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:49.387683 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:49.401531 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:49.402757 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:49.406786 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:49.413218 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:49.422410 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:49.422622 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:49.422806 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:49.485713 21522 consensus_queue.cc:237] T ec2a46f8cfb3406082970e5fdb59b101 P 778c78105092428bbcde75ab2cdb6a05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:10:49.487227 21136 raft_consensus.cc:1261] T ec2a46f8cfb3406082970e5fdb59b101 P 5bf81f3d986e428190d3f7727f36d96c [term 2 FOLLOWER]: Refusing update from remote peer 778c78105092428bbcde75ab2cdb6a05: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 14:10:49.487658 19375 raft_consensus.cc:1261] T ec2a46f8cfb3406082970e5fdb59b101 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 2 FOLLOWER]: Refusing update from remote peer 778c78105092428bbcde75ab2cdb6a05: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 14:10:49.488063 30684 consensus_queue.cc:1035] T ec2a46f8cfb3406082970e5fdb59b101 P 778c78105092428bbcde75ab2cdb6a05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:10:49.488257 29623 consensus_queue.cc:1035] T ec2a46f8cfb3406082970e5fdb59b101 P 778c78105092428bbcde75ab2cdb6a05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:10:49.490427 29623 raft_consensus.cc:2932] T ec2a46f8cfb3406082970e5fdb59b101 P 778c78105092428bbcde75ab2cdb6a05 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:49.493225 19375 raft_consensus.cc:2932] T ec2a46f8cfb3406082970e5fdb59b101 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:49.495718 21136 raft_consensus.cc:2932] T ec2a46f8cfb3406082970e5fdb59b101 P 5bf81f3d986e428190d3f7727f36d96c [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:49.495893 19225 catalog_manager.cc:5484] T ec2a46f8cfb3406082970e5fdb59b101 P fb1e6bc2d07b4ce3b622a66f14c93bdc reported cstate change: config changed from index 6 to 8, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New cstate: current_term: 2 leader_uuid: "778c78105092428bbcde75ab2cdb6a05" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:10:49.501041 19748 tablet_service.cc:1514] Processing DeleteTablet for tablet ec2a46f8cfb3406082970e5fdb59b101 with delete_type TABLET_DATA_TOMBSTONED (TS da3c4e948ff747678f18456c134d7ed4 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:36530
I20240501 14:10:49.510160 30705 tablet_replica.cc:330] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4: stopping tablet replica
I20240501 14:10:49.510428 30705 raft_consensus.cc:2229] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:49.510653 30705 raft_consensus.cc:2258] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:49.511430 30705 ts_tablet_manager.cc:1875] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:49.513975 30705 ts_tablet_manager.cc:1888] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240501 14:10:49.514058 30705 log.cc:1196] T ec2a46f8cfb3406082970e5fdb59b101 P da3c4e948ff747678f18456c134d7ed4: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal/wals/ec2a46f8cfb3406082970e5fdb59b101
I20240501 14:10:49.514710 19220 catalog_manager.cc:4830] TS da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357): tablet ec2a46f8cfb3406082970e5fdb59b101 (table rebalance_test_table_3 [id=35d6f8fac5394197b987aa44dab206c1]) successfully deleted
I20240501 14:10:49.529278 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:49.530361 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:49.530381 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:49.531589 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:49.533793 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:49.551633 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:49.552482 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:49.635349 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:49.636466 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:49.637144 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:49.637528 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:49.648685 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:49.663594 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:49.666086 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:10:49.695386 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:49.728958 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:10:49.761076 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:49.766018 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:49.769330 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:49.771598 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:49.779767 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:49.783335 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:10:49.800254 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:10:49.802601 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:49.891144 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:49.898342 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:49.900910 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:49.905035 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:49.911111 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:49.915266 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:49.923552 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:49.955933 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:49.956233 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:49.966744 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:49.972649 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:49.977064 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:49.980072 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:10:49.984331 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:49.989382 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:10:49.993849 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:10:50.083495 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:50.088524 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:10:50.091562 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:50.094003 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:50.094725 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:50.102806 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:50.111449 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:50.115581 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:50.126605 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:10:50.127821 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:10:50.130316 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:50.132381 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:50.139266 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:10:50.141111 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:10:50.143199 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:50.171686 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:50.190654 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:50.192080 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:10:50.239890 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:50.246830 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:10:50.274715 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:50.276515 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:50.281826 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:50.282639 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:50.287972 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:50.288223 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:50.304445 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:10:50.365057 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:50.374790 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:10:50.399843 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:50.405328 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:50.406230 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:50.412575 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:50.426864 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:50.428632 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:50.430040 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:10:50.504601 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:10:50.523648 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:50.529837 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:50.533996 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:50.536631 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:50.539301 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:50.543556 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:50.564666 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:50.619937 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:50.619930 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:50.621853 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:50.632073 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:50.643426 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:50.652112 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:50.660887 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:50.710386 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:50.713768 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:50.713850 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:10:50.724267 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:10:50.727702 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:50.731662 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:50.733789 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:50.759816 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:50.806299 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:50.808723 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:50.809134 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:50.809335 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:50.813378 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:50.844408 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:50.844612 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:50.938210 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:50.941843 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:50.944383 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:50.947659 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:50.959666 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:50.973429 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:50.975149 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:51.075627 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:51.076756 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:51.081153 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:51.092618 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:51.093384 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:51.097194 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:51.097195 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:51.104439 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:51.107563 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:51.119231 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:51.127883 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:51.128057 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:51.135066 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:51.145494 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:51.252029 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:51.253217 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:51.253926 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:51.260365 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:51.267267 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:51.282047 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:51.288019 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:10:51.316579 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:10:51.460829 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:51.461776 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:51.464646 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:51.472700 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:51.473987 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:51.479475 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:51.487628 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:10:51.492774 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:51.566387 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:10:51.591506 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:51.601240 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:51.611366 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:51.612725 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:51.638057 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:51.642573 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:51.647749 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:51.650422 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:51.669037 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:51.681772 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:51.683753 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:51.687172 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:51.688506 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:51.689159 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:10:51.777889 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:10:51.808239 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:51.813000 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:51.813306 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:51.822503 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:51.824601 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:51.842428 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:51.847605 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:51.873654 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:51.874635 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:51.882565 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:51.892184 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:51.895315 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:51.899524 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:51.910985 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:10:51.972602 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:10:51.993696 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:51.994683 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:51.995208 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:51.997474 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:51.998765 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:52.001830 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:52.003386 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:52.102619 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:52.108752 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:52.111687 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:52.117565 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:52.118568 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:52.134567 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:52.140937 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:52.217448 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:52.223417 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:52.225131 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:52.226023 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:10:52.227201 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:10:52.231688 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:10:52.233891 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:10:52.233964 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:52.258862 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:10:52.261899 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:10:52.339684 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:52.340060 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:52.340384 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:52.341420 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:52.353945 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:52.354485 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:52.361451 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:10:52.424456 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:10:52.456295 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:52.460155 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:52.461500 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:52.468693 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:52.471081 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:52.475520 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:52.490073 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:10:52.514026 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:10:52.586354 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:52.586730 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:52.588490 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:52.592402 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:52.593343 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:52.599850 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:52.602602 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:10:52.607955 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:52.639899 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:52.676344 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:52.749398 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:10:52.754933 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:10:52.758256 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:52.758852 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:52.759421 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:52.762220 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:52.768121 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:52.769600 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:52.789304 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:52.797024 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:52.799507 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:52.811189 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:52.819830 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:52.825291 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:52.829265 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:52.829495 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:10:52.878870 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:10:52.880043 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:52.939378 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 33 similar messages]
W20240501 14:10:52.942013 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:10:52.960171 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:52.973901 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:52.985445 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:52.986109 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:52.987665 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:53.004279 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:10:53.004932 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 14:10:53.005894 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:10:53.009778 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:53.010443 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:53.017709 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:53.014914 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:53.023363 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:53.038519 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:10:53.047356 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:10:53.049986 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:53.050992 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:10:53.066610 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240501 14:10:53.150622 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:10:53.172101 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:53.178112 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:53.178798 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:53.190029 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:53.207557 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:53.215436 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:53.216761 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:53.371520 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:53.372825 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:53.374567 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:53.379609 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:53.381786 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:53.399071 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:53.399488 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:53.489065 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:53.490110 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:53.494102 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:53.499547 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:53.506697 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:53.532857 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:53.545194 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:53.550518 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:53.555289 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:53.556778 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:53.574867 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:53.587729 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:53.600337 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:53.605357 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:10:53.621685 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:10:53.661114 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:10:53.691226 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:53.694943 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:53.700449 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:53.707697 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:53.707996 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:53.710500 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:53.725471 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:53.815927 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:53.820149 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:53.822247 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:53.824460 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:53.830598 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:53.848912 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:53.851783 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:53.909741 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:53.912679 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:53.919379 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:53.919615 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:53.923220 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:53.951471 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:53.951802 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:54.006608 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:54.014559 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:54.019412 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:54.020005 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:54.022114 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:54.025348 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:54.035086 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:10:54.070362 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:10:54.080163 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:10:54.087906 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:54.088776 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:54.093115 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:54.093219 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:54.095971 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:54.111356 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:54.123189 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:10:54.129509 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:10:54.191588 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:54.195950 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:54.196130 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:54.216444 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:54.216730 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:54.227806 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:54.230372 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:54.419288 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:54.419487 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:54.419878 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:54.420473 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:54.423718 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:10:54.435775 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:54.441097 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:54.443307 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:10:54.479941 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:10:54.554203 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:54.554235 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:54.555320 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:54.568478 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:54.576503 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:54.580358 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:54.580363 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:54.607595 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:54.614318 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:54.615064 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:54.618214 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:54.623492 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:54.640527 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:54.643977 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:10:54.664777 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:10:54.729190 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:54.730386 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:54.734048 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:54.744115 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:54.745039 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:54.751852 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:54.762570 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:54.846271 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:54.849802 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:54.851991 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:54.855201 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:54.858691 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:54.871218 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:54.872870 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:10:54.901667 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:54.916035 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:54.919992 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:10:54.970335 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:54.971443 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:54.976321 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:54.989632 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:54.998739 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:54.999719 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:55.001140 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:55.034809 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:55.037292 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:55.039175 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:55.039337 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:55.046483 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:55.052078 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:55.061836 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:55.144783 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:55.146701 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:55.154465 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:55.157251 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:55.158130 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:55.159602 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:55.160362 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:10:55.212292 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:10:55.227151 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:55.229887 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:10:55.230069 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:10:55.233299 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:55.233887 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:55.234457 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:55.234771 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:10:55.248778 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:10:55.249563 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:10:55.257023 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:10:55.304328 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:55.306203 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:55.357682 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:10:55.489789 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:55.490931 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:55.496388 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:55.500036 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:55.509441 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:55.519208 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:55.531862 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:10:55.531842 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:10:55.603806 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:55.607271 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:55.612694 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:55.613659 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:55.615895 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:55.618768 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:55.646871 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:55.672536 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:55.681746 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:55.682019 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:55.684401 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:55.693730 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:55.698017 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:55.705024 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:10:55.713313 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:10:55.806653 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:55.811790 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:55.814880 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:55.820029 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:55.824223 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:55.828641 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:55.834273 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:55.929952 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:55.930281 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:55.935600 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:55.936277 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:55.936373 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:55.966084 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:55.966709 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:56.063357 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:56.069464 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:56.073089 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:56.075387 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:56.077567 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:56.096462 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:56.097301 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:56.098002 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:56.106451 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:56.109747 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:56.112557 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:56.130679 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:56.131982 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:56.143601 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:56.232468 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:56.242059 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:56.242583 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:56.249027 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:56.252810 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:56.253477 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:56.260668 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:56.265231 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:10:56.292979 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:10:56.315120 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:56.316222 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:56.316299 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:56.317997 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:56.320641 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:56.322436 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:56.331146 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:10:56.465222 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:10:56.577520 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:56.578333 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:56.579124 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:56.582702 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:56.593977 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:56.603343 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:56.620651 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:56.702392 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:56.707201 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:56.708757 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:56.734800 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:56.736366 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:56.738888 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:56.740417 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:10:56.758127 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:10:56.759229 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:56.763126 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:56.767074 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:56.778439 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:56.778550 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:56.778896 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:56.805972 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:10:56.809692 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:10:56.897008 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:56.897820 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:56.899498 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:56.901451 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:56.907790 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:56.924160 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:56.932617 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:57.017886 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:57.020263 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:57.021041 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:57.021667 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:57.025527 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:57.027729 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:57.042304 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:57.126719 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:57.128198 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:57.129542 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:57.146749 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:57.151414 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:57.154929 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:57.168845 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:57.204399 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:57.209219 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:57.213449 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:57.214418 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:57.221194 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:57.225111 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:57.227393 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:10:57.252923 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:57.280614 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:10:57.304967 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:57.308846 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:57.309459 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:57.313125 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:57.314230 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:57.315187 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:57.315725 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:57.360037 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:10:57.390767 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:57.395834 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:57.398176 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:57.398751 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:57.398778 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:57.400804 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:57.401456 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:10:57.546838 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:57.610069 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:57.625124 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:10:57.644425 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:57.645957 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:57.651839 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:57.656343 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:57.673920 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:57.677848 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:57.694137 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:10:57.707912 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:57.721854 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:57.749214 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:10:57.757639 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:57.757918 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:57.758852 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:57.761634 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:57.777966 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:57.778681 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:57.787539 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:10:57.836023 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:10:57.848763 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:57.855165 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:57.856576 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:57.859578 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:57.860127 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:57.863628 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:10:57.868155 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:10:57.889839 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:10:57.936381 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:57.941249 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 14:10:57.942690 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:10:57.942975 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:10:57.949105 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:57.953405 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:57.954083 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:57.967777 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:57.982000 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:57.984328 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:58.070288 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:58.073311 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:58.073333 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:58.078413 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:58.081063 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:10:58.083909 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240501 14:10:58.086560 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:58.091687 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:58.203070 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:58.206895 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:58.208626 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:10:58.225091 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:10:58.225729 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:58.226987 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:58.232283 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:58.233950 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:10:58.241487 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:10:58.251828 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:10:58.252100 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:10:58.253110 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:58.253118 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:58.260406 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:58.262761 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:58.267798 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:58.301149 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240501 14:10:58.305402 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:58.379914 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:58.387022 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:58.387277 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:58.387872 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:58.390198 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:58.402241 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:58.415307 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:58.449105 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:58.455687 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:58.458871 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:58.472885 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:58.472879 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:58.489585 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:58.491252 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:58.719516 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:10:58.743558 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:58.744226 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:58.748147 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:58.759523 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:58.761955 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:58.762377 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:58.767007 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:58.842105 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:58.846212 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:58.849130 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:58.856738 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:58.861418 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:10:58.864995 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:10:58.871176 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:58.872112 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:10:58.874442 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:58.874528 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:58.928526 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:58.933183 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:58.937188 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:58.942510 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:58.951820 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:58.952914 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:58.960681 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:59.034287 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:59.036222 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:59.044445 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:59.050604 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:59.055352 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:59.057516 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:59.076589 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:59.111752 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:59.112354 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:59.117753 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:59.118105 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:59.122326 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:59.129392 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:59.131740 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:10:59.218628 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:10:59.229737 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:59.230450 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:59.242359 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:59.243117 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:59.243501 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:59.246780 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:59.265683 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:10:59.315310 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:10:59.358644 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:59.358935 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:59.359369 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:59.360895 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:59.371127 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:59.397483 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:59.402035 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:59.441013 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:59.442780 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:59.443202 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:59.452486 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:59.453254 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:59.458076 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:59.468972 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:59.556020 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:59.557586 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:59.565057 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:59.566646 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:59.571728 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:59.578850 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:59.587247 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:59.697469 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:59.699512 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:59.703119 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:10:59.704905 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:59.705209 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:59.705816 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:59.710430 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:10:59.737188 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:59.762396 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:10:59.778463 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:10:59.882030 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:59.887212 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:10:59.890110 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:59.898703 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:10:59.900545 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:10:59.901508 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:10:59.922147 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:10:59.926255 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:10:59.974695 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:11:00.058423 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:11:00.114926 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:11:00.173892 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:11:00.200232 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:11:00.242327 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:11:00.248308 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:11:00.297530 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:11:00.602471 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:00.768713 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:00.772836 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:00.777695 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:00.779103 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:00.783074 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:00.787843 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:00.804270 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:11:00.813088 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:00.894227 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:00.897683 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:00.898934 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:00.900570 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:00.904178 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:00.927827 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:00.939885 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:00.998553 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:01.002067 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:01.002879 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:01.003926 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:01.007099 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:01.020781 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:01.022308 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:01.023814 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:01.083925 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:01.086829 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:01.087546 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:01.091837 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:01.104182 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:01.107524 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:01.117671 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:01.126483 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:11:01.205277 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:01.208303 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:01.208598 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:01.210846 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:01.215523 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:01.226992 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:01.234632 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:01.321810 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:01.322419 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:01.323196 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:01.327498 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:01.328392 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:01.329591 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:01.371731 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:01.448596 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:01.450199 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:01.452028 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:01.455456 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:01.456542 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:01.472522 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:01.479149 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:01.514772 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:11:01.556305 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:01.560120 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:01.560655 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:01.570827 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:01.576377 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:01.579378 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:01.580801 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:11:01.648718 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:11:01.659317 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:11:01.677893 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:01.681469 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:01.684605 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:01.694314 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:01.697732 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:01.704021 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:01.717092 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:01.823014 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:01.823957 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:01.824684 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:01.833369 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:01.855788 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:01.861769 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:01.866256 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:01.866608 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:01.873567 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:01.874662 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:01.879139 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:01.879314 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:01.904670 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:01.906761 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:01.996358 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:01.999652 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:02.007256 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:02.013051 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:02.014436 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:02.027230 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:02.029935 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:11:02.035406 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:02.075716 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:02.081239 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:02.085431 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:02.086222 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:02.087239 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:02.091230 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:02.099558 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:02.195530 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:02.196821 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:02.205341 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:11:02.208011 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:02.209782 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:02.211480 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:02.223423 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:02.227597 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:02.325588 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:11:02.334333 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:02.338865 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:02.342693 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:02.344345 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:02.351765 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:02.367447 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:02.368347 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:02.412298 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:11:02.455885 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:02.458241 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:02.459154 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:11:02.460498 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:11:02.464463 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:02.466094 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:02.475525 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:02.476310 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:02.482223 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:02.504333 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:11:02.576975 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:02.577224 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:02.583252 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:02.584475 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:02.586066 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:02.591198 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:02.593436 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:02.600455 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:02.633234 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:11:02.702071 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:02.706291 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:02.706331 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:02.708534 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:02.711131 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:02.739836 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:02.742060 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:02.748629 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:11:02.825549 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:02.838120 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:02.844426 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:02.860494 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:02.868746 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:02.869221 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:02.870775 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:02.873052 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:02.882697 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:02.884053 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:02.886807 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:02.889444 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:02.889915 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:02.909411 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:11:02.936161 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:02.966619 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:11:03.007414 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:11:03.009898 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:03.019224 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:03.023166 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:03.029567 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:03.031698 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:03.044821 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:03.045138 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:03.056658 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:11:03.058475 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:03.088351 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:03.090829 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 72 similar messages]
I20240501 14:11:03.095466 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:03.096328 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:03.106355 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:03.106534 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:03.108747 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:03.115767 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:03.215184 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:03.215644 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:03.219705 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:03.224399 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:03.229445 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:03.241605 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:03.243505 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:03.388084 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:03.388221 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:03.388696 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:03.391403 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:03.395228 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:03.402730 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:03.403589 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:03.405144 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:03.502137 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:03.507700 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:03.509217 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:03.515851 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:03.518558 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:03.523869 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:03.554365 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:03.585047 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:11:03.595980 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:03.597028 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:03.597813 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 14:11:03.597921 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:11:03.599639 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:11:03.603197 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:03.608269 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:03.610508 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:03.611004 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:03.716661 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:03.720883 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:03.722285 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:03.726014 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:03.734280 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:03.739248 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:03.762727 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:03.820149 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:11:03.858783 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:03.860252 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:03.867090 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:03.882607 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:03.884080 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:03.884322 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:03.898576 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:03.899729 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:03.900079 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:03.911691 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:03.911897 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:11:03.922045 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:03.923111 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:03.927228 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:03.931764 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:03.968940 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 14:11:03.970469 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:11:04.052978 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:04.061758 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:04.063299 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:04.066306 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:04.067350 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:04.069931 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:04.071616 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:04.072275 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:04.082195 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:04.088037 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:04.091363 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:04.103214 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:04.120677 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:04.148670 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:04.231166 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:04.231263 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:04.235195 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:04.237766 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:04.238247 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:04.251071 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:04.251899 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:11:04.253703 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:04.416718 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:04.417672 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:04.418960 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:04.420997 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:04.421761 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:04.427361 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:04.449505 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:04.481021 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:11:04.531715 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:11:04.532871 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:04.534744 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:04.535531 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:04.537384 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:04.544862 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:04.565304 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:04.565753 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:04.842203 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:11:04.858425 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:04.858727 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:04.867322 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:04.869524 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:04.877991 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:04.885838 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:04.886076 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:04.886142 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:04.909137 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:11:04.928123 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:11:04.981859 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:04.984045 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:04.987185 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:04.991369 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:05.006093 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:05.007580 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:05.021764 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:11:05.026085 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:11:05.052163 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:11:05.146874 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:11:05.193665 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:11:05.311304 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:05.311817 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:05.313452 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:05.319082 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:05.320581 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:05.327468 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:05.330534 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:05.414839 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:05.427925 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:05.433001 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:11:05.433822 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:05.435400 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:05.437047 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:05.439903 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:05.447650 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:05.447898 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:05.544716 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:05.550560 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:05.551674 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:05.555729 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:05.556090 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:05.559463 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:05.563982 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:11:05.573158 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:05.593255 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:11:05.673642 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:05.675837 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:05.677050 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:05.678977 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:05.679983 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:05.680536 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:05.710296 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:05.797838 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:05.800037 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:05.801471 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:05.803267 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:05.803584 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:05.803686 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:05.808482 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:05.812717 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:05.882539 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:05.885226 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:05.888489 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:05.891845 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:05.895565 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:05.904691 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:05.922412 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:05.951086 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:11:05.983898 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:05.987735 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:05.995640 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:05.996452 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:06.016032 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:06.021544 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:06.031755 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:06.077296 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:06.080803 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:06.084390 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:06.086081 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:06.087220 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:06.089644 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:06.117280 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:06.205997 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:06.211256 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:06.213799 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:06.214635 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:06.216269 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:06.227249 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:06.228294 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:06.318513 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:06.323650 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:06.324331 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:06.329865 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:06.332376 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:06.336198 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:06.339143 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:11:06.361263 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:06.369674 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:06.435930 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:06.440505 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:06.441203 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:06.442173 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:06.443332 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:06.445477 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:06.470103 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:06.545261 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:06.546103 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:06.551937 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:06.555166 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:06.555743 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:06.569169 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:06.587203 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:06.674620 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:06.679163 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:06.682005 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:06.682237 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:06.687252 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:06.689075 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:06.695466 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:06.765810 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 14:11:06.792402 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:06.793603 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:06.797700 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:06.809607 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:06.809839 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:06.819562 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:06.819658 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:06.821350 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:06.858834 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:06.858850 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:06.859299 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:06.863760 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:06.873776 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:06.890331 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:06.891001 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:06.931552 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 14:11:06.979346 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:06.980919 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:06.981498 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:06.981765 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:06.995481 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:06.995739 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:07.020295 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:07.049527 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:07.051765 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:07.054380 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:07.054863 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:07.060256 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:07.071578 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:07.081681 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:07.147362 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 14:11:07.163756 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:07.165551 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:07.167402 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:07.167781 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:07.185370 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:07.195225 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:07.207150 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:07.277864 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 14:11:07.336881 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:07.338114 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:07.339004 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:07.343634 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:07.348686 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:07.349328 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:07.364291 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:07.380810 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:07.443317 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 14:11:07.462347 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:07.464007 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:07.466585 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:07.471163 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:07.476799 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:07.493203 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:07.505559 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:07.532335 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 14:11:07.556176 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:07.559216 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:07.565687 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:07.567214 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:07.570441 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:07.574717 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:07.575452 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:07.587002 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:11:07.676026 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 14:11:07.687186 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:07.689095 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:07.689661 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:07.691545 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:07.692862 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:07.707248 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:07.721693 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:07.724094 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:07.819003 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:07.835098 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:07.836254 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:07.837373 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:07.838477 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:07.841542 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:07.843370 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:07.843689 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:07.864862 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:07.878505 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:07.879963 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:07.881801 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:07.882952 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:07.883843 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:07.926505 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 14:11:08.024505 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:08.024914 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:08.032644 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:08.036677 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:08.037093 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:08.045934 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:08.056819 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:08.060931 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:08.075196 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:08.079031 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:08.087538 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:08.089756 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:08.098825 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:08.114557 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:08.129595 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 14:11:08.134531 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:11:08.144910 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:11:08.145543 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:11:08.179960 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:08.198419 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:08.200167 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:08.212498 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:08.213414 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:08.213677 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:08.223356 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:08.228407 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:08.390652 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 14:11:08.396807 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:08.398936 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:08.400585 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:08.402129 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:08.407543 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:08.407563 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:08.424276 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:08.509560 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 14:11:08.515931 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:08.516471 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:08.522846 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:08.525656 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:08.527742 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:08.540899 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:08.563606 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:08.564445 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:08.594422 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:08.594977 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:08.605322 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:08.607111 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:08.618357 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:08.618989 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:08.620180 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:08.668783 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:11:08.717914 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:08.719619 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:08.722358 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:08.731474 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:08.731613 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:08.734174 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:08.756232 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:08.841385 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:08.842530 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:08.848915 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:08.861542 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:08.868942 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:08.874122 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:08.879930 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:08.896363 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 14:11:08.933894 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:08.936558 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:08.937559 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 14:11:08.938732 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:08.942603 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:08.942935 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:08.943178 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:08.951444 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:11:08.954365 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:09.035096 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:09.037474 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:09.044023 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:09.049458 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:09.051975 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:09.056645 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:09.059208 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:09.069080 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:09.111305 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:09.113648 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:09.116287 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:09.117805 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:09.128510 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:09.129138 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:09.129925 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:09.208750 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:09.213604 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:09.216631 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:09.218210 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:09.223979 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:09.230319 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:09.231015 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:09.268833 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:11:09.373319 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 14:11:09.432426 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:09.435321 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:09.441577 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:09.450646 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:09.453351 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:09.453505 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:09.469688 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:09.473918 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:11:09.503221 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:11:09.559032 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:09.561425 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:09.576272 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:09.577204 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:09.588474 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:09.598052 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:09.604215 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:09.604426 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:09.605015 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:09.611879 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:09.626150 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:09.641122 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:09.646005 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:09.653486 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:09.697104 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:11:09.723835 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 14:11:09.753448 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:09.756552 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:09.758672 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:09.758764 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:09.759037 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:09.774441 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:09.785861 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:09.853580 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 14:11:09.868853 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:09.871630 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:09.871902 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:09.881939 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:09.883632 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:09.885201 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:09.894532 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:09.936473 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:11:09.964196 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 14:11:10.006623 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:10.012804 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:10.013432 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:10.017655 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:10.023245 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:10.025506 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:10.033236 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:10.040728 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:11:10.042273 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 14:11:10.046473 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:10.048043 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:10.053897 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:10.074234 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:10.074491 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:10.095173 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:10.182286 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:10.198505 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:10.200105 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:10.203114 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:10.211334 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:10.219193 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 14:11:10.224753 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:10.225013 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:10.242282 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:10.246768 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:10.257979 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:10.258796 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:10.259783 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:10.262902 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 14:11:10.274917 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:10.295336 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:10.524716 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:10.598942 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:10.602005 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:10.606834 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:10.614701 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:10.615024 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:10.626000 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:10.639104 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:10.640137 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:10.663669 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:10.664547 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:10.670475 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:10.678684 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:10.682741 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:10.690649 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 14:11:10.692242 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:10.780853 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:10.785264 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:10.786002 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:10.789958 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:10.791675 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:10.796583 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:10.817679 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:10.851200 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:10.852824 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:10.857816 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:10.871565 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:10.872380 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:10.885100 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:10.887476 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:10.904126 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 14:11:10.967259 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:10.967370 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:10.973040 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:10.973906 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:10.981684 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:10.992702 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 14:11:10.998231 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:10.998554 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:11.012964 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:11.087452 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:11.091370 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:11.092271 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:11.104475 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:11.106132 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:11.113811 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 14:11:11.115746 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:11.116761 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:11.235257 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:11.238402 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:11.239162 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:11.242750 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:11.247278 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:11.247716 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:11.269985 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:11.272323 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:11.276309 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:11.289907 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:11.291158 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:11.302677 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:11.328780 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:11.334724 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:11.409868 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:11.411092 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:11.416913 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:11.421149 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:11.426810 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:11.427126 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:11.432132 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:11:11.439111 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:11:11.505007 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 14:11:11.626633 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:11.628129 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:11.637761 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:11.638224 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:11.638479 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:11.647471 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:11.648864 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:11.682571 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 14:11:11.766577 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:11.774994 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:11.775441 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:11.780005 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:11.788504 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:11.792760 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:11.795146 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:11.802520 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:11.815760 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:11.822731 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:11.832756 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:11.840812 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:11.844434 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:11.860680 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:11.878856 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:11.913691 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 14:11:11.980630 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:11.984690 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:11.987231 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:11.988785 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:11.996109 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:11.999202 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:12.006058 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:12.007956 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:12.010778 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:12.013479 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:12.031558 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:12.039273 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:12.059342 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 14:11:12.064885 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:12.069089 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:12.144745 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:12.146080 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:12.146910 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 14:11:12.152065 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:12.152684 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:12.156328 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:12.168795 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:12.170229 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:12.267262 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:12.273960 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:12.277760 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:12.281733 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:12.283175 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:12.285399 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:12.291267 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:12.365986 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:11:12.372337 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:12.380738 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:12.381446 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:12.386188 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:12.390429 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:12.392762 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:12.395675 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:12.423172 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:12.433753 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 14:11:12.498925 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:12.499449 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:12.501463 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:12.504914 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:12.512445 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:12.513518 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:12.520545 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:12.617650 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:12.619396 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:12.625406 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:12.627221 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:12.628350 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:12.647576 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:12.664247 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:12.678561 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:11:12.720259 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 14:11:12.754567 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:12.756981 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:12.761675 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:12.763957 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:12.765873 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:12.768090 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:12.778857 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:11:12.807274 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 14:11:12.888242 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:12.893111 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:12.894165 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:12.898773 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:12.913524 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:12.915719 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:12.917243 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:12.925604 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:12.926189 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:12.941498 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:12.943076 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:11:12.951753 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:12.959005 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:12.973726 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:12.988788 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:13.060132 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:13.066212 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:13.068992 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:13.076089 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:13.079872 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:13.081619 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:13.082942 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:13.136601 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:13.136843 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:13.137450 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:13.137739 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:13.147792 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:13.149228 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:13.166608 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:11:13.217013 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240501 14:11:13.254526 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:13.258162 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:13.258831 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:13.260022 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:13.268338 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:13.279302 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 14:11:13.280402 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:13.292251 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:13.299325 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:11:13.300348 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 14:11:13.422475 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:13.423962 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:13.428476 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:13.433151 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:13.441190 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:13.445741 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:13.458837 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:13.528445 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:13.544966 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:13.545992 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:13.546479 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:13.559262 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:13.570127 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:13.577298 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:13.588027 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:13.637630 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:13.641307 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:13.641749 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:13.643561 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:13.644850 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:13.653640 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:11:13.663867 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:11:13.677628 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 14:11:13.680258 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:13.768985 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:13.772143 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:13.777248 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:13.777237 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:13.778453 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:13.779345 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:13.787639 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:13.900378 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:13.903842 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:13.904634 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:13.904790 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:13.918583 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:13.921046 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:13.921188 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:13.932351 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:13.936790 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:13.950634 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:13.950807 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:13.954121 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:13.954231 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:13.965279 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:13.967599 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:14.058548 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:11:14.059362 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:11:14.062144 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 14:11:14.104892 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:14.107766 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:14.126232 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:14.130831 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:14.136049 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:14.138208 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:14.140291 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:14.158236 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:14.160669 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:14.184458 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:14.188931 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:14.207609 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:14.213462 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:11:14.238415 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 14:11:14.258220 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:14.347194 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:14.348722 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:14.352661 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:14.356025 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:14.358078 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:14.360100 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:14.388550 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:14.395407 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:11:14.518033 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:14.522541 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:14.523175 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:14.527439 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:14.530421 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:14.534934 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:14.535135 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:14.549680 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:14.646421 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:14.650097 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:14.655100 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:14.660128 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:11:14.665752 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 14:11:14.667496 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:14.673166 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:14.673262 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:14.686539 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:11:14.704911 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 14:11:14.748747 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:14.752066 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:14.753849 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:14.754302 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:14.754706 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:14.771544 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:14.772933 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:14.796083 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 14:11:14.861796 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 14:11:14.865069 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:14.869410 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:14.871941 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:14.875979 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:14.876518 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:14.893391 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:14.897153 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:14.903795 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 14:11:14.989897 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:14.991505 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:14.992580 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:15.003338 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:15.005043 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:15.006207 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:15.024855 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:15.030179 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:15.031069 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 14:11:15.034063 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:15.054415 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:15.056938 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:15.068619 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:15.070154 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:15.091728 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:15.096519 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:11:15.136726 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 14:11:15.167299 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:15.176069 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:15.183393 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:15.195909 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:15.196882 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:15.206399 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:15.210378 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:15.233574 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:15.234628 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:15.238068 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:15.248975 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:15.261394 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:15.263795 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:15.266680 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:15.367398 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:15.367588 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:15.368721 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:15.373842 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:15.390565 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:15.391315 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:15.405843 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:15.424140 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:11:15.462364 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 14:11:15.577636 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:15.580562 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:15.581372 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:15.583241 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:15.585991 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 14:11:15.589613 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:15.591465 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:15.616631 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:15.644332 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 14:11:15.716174 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:15.716349 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:15.723312 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:15.723441 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:15.736514 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:15.740446 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:15.738732 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:15.742609 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:15.756150 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:15.779626 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:15.788759 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:15.789441 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:15.791285 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:15.802372 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:15.896541 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:15.897723 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:15.899545 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:15.900200 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:15.914324 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:15.924638 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:15.926963 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:15.990039 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:16.004097 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:16.009510 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:16.010538 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:16.018002 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:16.021876 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:16.027715 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:16.049618 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:11:16.052110 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:11:16.108052 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 14:11:16.130137 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:16.130826 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:16.134816 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:16.136925 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:16.152328 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:16.152716 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:16.159212 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:16.193342 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:16.198973 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:16.199684 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:16.224689 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:16.224800 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:16.227758 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:16.230279 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:16.312386 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:16.312422 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:16.316766 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:16.317585 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:16.319240 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:16.324510 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:16.334264 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:16.398900 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:16.400009 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:16.402956 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:11:16.407006 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:16.408942 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:16.409794 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:16.412616 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:16.434674 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:16.640473 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:16.644259 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:16.650856 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:16.650951 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:16.658362 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:16.664408 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:16.677698 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:16.683334 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:11:16.729034 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 14:11:16.766417 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:16.768039 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:16.774300 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:16.778957 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:16.781309 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:16.784493 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:16.801324 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:16.829720 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:16.835350 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:16.836576 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:16.846712 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:16.847635 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:16.864173 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:16.864782 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:16.902513 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:11:16.930495 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 14:11:16.954943 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:16.956745 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:16.958529 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:16.967588 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:16.969063 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:16.988535 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:16.995719 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:17.007112 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 14:11:17.077385 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:17.079635 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:17.080315 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:17.081059 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:17.094722 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:17.096572 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:17.108855 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:17.225543 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:17.229923 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:17.232865 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:17.233256 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:17.238536 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:17.241576 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:17.244441 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:17.246410 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:17.259104 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:17.259562 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:17.271121 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:17.279320 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:17.288049 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:17.291738 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 14:11:17.293812 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:17.294772 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:17.384637 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 14:11:17.408555 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:17.408789 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:17.426975 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:17.427428 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:17.429020 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:17.431932 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:17.445740 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:17.454339 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:17.459497 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:17.463446 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:17.470407 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:17.477200 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:17.486649 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:17.496958 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:11:17.509395 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:11:17.667439 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:11:17.725764 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 14:11:17.810065 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:17.813428 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:17.818836 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:17.819726 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:17.831421 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 14:11:17.832118 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:17.832240 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:17.838099 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:17.851126 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:17.851217 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:17.857120 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:17.874423 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:17.880785 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:17.903067 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:17.908722 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:17.914563 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:17.973747 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:11:17.988621 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 14:11:17.996459 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:18.005504 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:18.005543 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:18.009523 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:18.014348 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:18.018873 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:18.035418 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:18.071802 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:18.075603 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:18.082618 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:18.085547 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:18.090308 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:18.098059 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:18.099066 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:18.194734 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:18.195349 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:18.206198 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:18.206504 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:18.208678 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:18.220974 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:18.225823 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:18.226774 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 68 similar messages]
I20240501 14:11:18.326371 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:18.327072 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:18.330116 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:18.334323 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:11:18.346346 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:11:18.347689 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:18.371206 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:18.371790 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:18.379556 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:18.393373 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:18.397646 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:18.406355 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:18.407613 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:18.409555 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:11:18.432533 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 14:11:18.439258 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:18.444787 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:18.512980 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:18.515254 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:18.517441 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:18.520345 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:18.530121 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:18.533663 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:18.559305 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:18.595261 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:18.599866 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:18.602205 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:18.606153 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:18.609483 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:18.614467 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:18.637370 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:18.716122 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 14:11:18.857702 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:18.859704 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:18.860906 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:18.861454 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:18.878712 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:18.888132 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:18.891639 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:18.958003 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:18.978991 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:18.985592 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:18.988823 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:19.001574 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:19.005198 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:19.022660 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:19.034536 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:19.036211 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:19.040764 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:19.041714 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:19.044787 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:19.049834 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:19.055212 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:19.083814 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:19.085180 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:11:19.162562 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:19.180804 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:19.181675 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:19.187158 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:19.194511 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:19.200942 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:19.204524 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:19.207500 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:19.210350 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:19.219714 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:19.221664 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:19.221884 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:19.239198 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 14:11:19.258627 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:11:19.259889 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 14:11:19.272264 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:19.311617 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 14:11:19.346029 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:19.357801 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:19.358781 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:19.359112 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:19.359568 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:19.370602 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:19.380014 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 14:11:19.380815 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:19.399618 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 14:11:19.409518 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:19.467568 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:19.470100 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:19.472292 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:19.475557 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:19.477967 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:19.486534 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:19.506323 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:19.531633 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 14:11:19.581004 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:19.585106 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:19.585193 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:19.591369 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:19.592789 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:19.599637 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:19.613360 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:19.697329 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:19.698041 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:19.700765 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:19.704021 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:19.704754 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:19.705672 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:19.709286 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:11:19.812616 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:19.813601 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:19.817373 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:19.817951 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:19.830153 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:19.830406 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:19.842650 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:19.852140 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:19.942399 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:19.944651 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:19.944684 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:19.945484 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:19.946255 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:19.950374 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:19.956308 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 14:11:19.971920 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:19.985997 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 14:11:20.070600 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:20.076334 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:20.076776 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:20.079337 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:20.085888 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:20.096355 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:20.098326 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:20.100936 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:20.114171 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:20.115420 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 14:11:20.124258 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:20.126199 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:20.156189 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:20.161247 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:20.167285 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:20.195712 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 14:11:20.253818 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:20.256876 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:20.257076 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:20.267133 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:20.269793 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:20.282284 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:20.290424 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:20.325420 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:20.325809 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:20.326267 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:20.327144 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:20.328038 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:11:20.340257 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:20.340847 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:20.353741 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:11:20.372328 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:11:20.399915 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:11:20.430588 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 14:11:20.445796 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:20.446370 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:20.451962 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:20.454249 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:20.454952 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:20.465078 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:20.470240 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:20.559468 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 14:11:20.609541 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:20.612226 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:20.614533 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:20.619091 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:20.624222 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:20.624465 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:20.625182 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:20.674181 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 14:11:20.741350 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:20.749222 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:20.755339 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:20.758569 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:20.768587 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:20.771191 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:20.772833 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:20.827313 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:20.830737 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:20.834334 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:20.839629 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:20.849954 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:20.856700 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:20.861562 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:20.884644 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:20.958995 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:20.962440 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:20.971417 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:20.972528 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:20.973886 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:20.982259 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:20.993686 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:11:21.048454 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:11:21.091300 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 14:11:21.096812 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:21.097355 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:21.098456 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:21.100059 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:21.107918 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:21.121670 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:21.130986 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:21.138851 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:21.138939 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:21.142143 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:21.148572 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:21.158735 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:21.165628 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:21.191383 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:21.288923 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:21.303269 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:21.304095 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:21.305179 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:21.308414 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:21.311811 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:21.316587 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:21.343825 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:21.345729 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:21.346535 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:21.359195 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:21.363536 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:21.371769 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:21.385658 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:21.401589 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:21.480993 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:21.481459 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:21.486251 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:21.487850 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:21.489445 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:21.501540 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:21.502295 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:21.647567 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:11:21.659456 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 14:11:21.665252 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:21.667371 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:21.670436 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:21.679021 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:21.679780 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:21.680033 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:21.693125 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:21.762430 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 14:11:21.791661 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:21.793337 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:21.794263 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:21.796953 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:21.797293 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:21.802373 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:21.804780 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:21.805614 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:21.873809 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:21.874294 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:21.876901 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:21.877625 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:21.881376 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:21.883436 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:21.901561 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:22.013703 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:22.014446 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:22.019373 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:22.019729 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:22.020756 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:22.039364 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:22.041935 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:22.140259 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:22.141369 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:22.144083 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:22.145902 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:22.151803 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:22.153640 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:22.164569 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:22.189388 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:22.190337 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:22.206044 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:22.214789 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:22.216784 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:22.220240 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:22.240718 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:22.265089 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 14:11:22.333571 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:22.333901 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:22.334025 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:22.345103 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 14:11:22.347627 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:22.353490 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:22.369701 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:22.376794 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:22.381567 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:22.391433 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:22.397336 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:22.413677 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:22.415186 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:22.417945 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:22.427942 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:22.431804 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:22.510713 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:22.511925 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:22.513242 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:22.514366 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 14:11:22.517184 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:22.529590 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:22.533701 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:22.543591 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:22.674253 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:22.674578 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:22.675125 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:22.675822 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:22.678874 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:22.680384 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:22.707777 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:22.728281 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 14:11:22.790685 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:22.794622 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:22.795452 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:22.802018 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:22.804100 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:22.811496 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:22.824460 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:22.891073 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:22.891640 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:22.893378 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:22.895833 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:22.900771 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:22.918399 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:22.922921 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:11:22.923847 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 14:11:22.928315 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:22.985147 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:11:22.993448 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 14:11:23.009649 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:23.012784 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:23.016739 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:23.027217 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:23.028448 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:23.029870 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:23.030118 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:23.050056 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 14:11:23.142345 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:23.149786 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:23.152653 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:23.159863 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:23.163349 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:23.169462 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:23.171597 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:23.173023 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:23.174386 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:23.194357 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:23.196193 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:23.197801 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:23.208032 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:23.244802 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:23.264508 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 14:11:23.278394 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 14:11:23.323745 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:23.324949 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:23.336171 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:23.337882 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:23.345412 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:23.359299 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:23.364336 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:23.385393 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:23.394639 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:23.408288 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:23.415164 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:23.415017 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:23.415453 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:23.426573 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:11:23.435019 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:11:23.473436 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 14:11:23.511838 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:23.511937 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:23.516794 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:23.519567 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:23.519788 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:23.524531 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:23.546061 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:23.547495 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:23.550685 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 14:11:23.677179 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:23.678010 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:23.683295 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:23.688839 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:23.688987 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:23.695233 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:23.711272 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:23.797550 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:23.798142 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:23.803107 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:23.813678 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:23.815419 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:23.816910 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:23.819633 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:23.901115 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:23.901788 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:23.903887 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:23.907819 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:23.910485 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:23.918186 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:23.921563 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:24.023790 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:24.025485 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:24.029467 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:24.030292 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:24.036408 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:24.043344 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:24.049962 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:24.071919 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:11:24.091971 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:24.174190 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:24.174399 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:24.186935 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:24.188417 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:24.196950 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:24.223534 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:24.238231 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:24.242084 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:24.248584 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:24.253779 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:24.253849 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:24.259382 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:24.269483 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 14:11:24.285495 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:24.307860 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:11:24.388703 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 14:11:24.413750 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:24.417014 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:24.425628 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:24.426064 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:24.434319 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:24.436875 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:24.444547 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 14:11:24.447100 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:24.475935 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:11:24.492439 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:24.492600 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:24.496698 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:24.509164 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:24.513461 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:24.513715 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:24.514369 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:24.533471 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:24.607496 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:24.607177 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:24.614523 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:24.619773 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:24.620601 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:24.627846 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:24.628758 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:11:24.638801 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 14:11:24.750959 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:24.756687 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:24.759124 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:24.759227 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:24.760214 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:24.773615 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:24.780181 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:24.795805 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 14:11:24.865240 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:24.868798 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:24.870154 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:24.872507 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:24.875237 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:24.875794 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:11:24.876206 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:24.894992 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:24.924909 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:24.980221 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:24.981942 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:24.984285 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:24.994626 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:24.996800 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:25.004757 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:25.007014 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:25.100113 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:25.100342 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:25.100981 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:25.101512 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:25.108040 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:25.117301 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:25.126245 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:25.201613 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:11:25.214284 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 14:11:25.215757 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:25.226454 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:25.230978 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:25.241191 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:25.244338 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:25.249823 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:25.258502 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 14:11:25.261184 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:25.265027 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:25.267714 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:25.270902 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:25.279914 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:25.298796 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:25.300320 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:25.306628 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:25.388754 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 14:11:25.415158 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:25.415485 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:25.418718 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:25.436012 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:25.446449 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:25.447088 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:25.447304 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:25.452664 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:25.458159 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:25.460338 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 14:11:25.461313 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:25.467360 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:25.473681 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:25.490190 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:25.518621 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:25.533536 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:11:25.542541 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 14:11:25.599773 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:25.603822 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:25.605182 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:25.606655 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:25.609835 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:25.615965 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:25.632378 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:25.709419 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:11:25.800602 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:25.803766 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:25.804802 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:25.807487 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:25.817065 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:25.823607 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:25.857746 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:25.915562 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 14:11:25.937628 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:25.942891 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:25.953948 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:25.956053 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:25.965633 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:25.968351 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:25.982656 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:25.986846 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:25.987722 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:26.004386 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:26.012125 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:26.022596 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:26.026567 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:26.029947 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 14:11:26.042110 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:26.119035 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:26.121562 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:26.121809 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:26.125607 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:26.127420 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:26.127672 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:26.131387 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:26.150211 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:26.201416 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 14:11:26.236105 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:26.236670 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:26.239212 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:26.240576 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:26.249539 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:26.253152 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:26.254261 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:26.360963 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:26.364275 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:26.378927 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:26.381350 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:26.387282 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:26.396183 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:26.396700 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:26.397461 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:26.409111 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:26.418689 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:26.419282 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:26.426946 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:26.443948 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:26.446206 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:26.502559 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 14:11:26.555361 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:26.559774 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:26.564244 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:26.568833 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:26.573966 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:26.577418 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:26.577672 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:26.586055 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:26.586447 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:26.588686 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:26.599224 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:26.608950 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:26.627463 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:26.627589 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:11:26.670419 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:26.955777 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:26.962391 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:26.962390 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 14:11:26.965113 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:26.970499 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:26.972100 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:26.974306 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:26.977754 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:26.978801 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:26.983310 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:26.988977 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 14:11:26.989771 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:26.998157 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:27.011824 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:27.021026 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:27.025209 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:27.033246 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:27.152170 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:27.161273 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:27.162436 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:27.164741 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:27.164974 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:27.178747 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:27.181670 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:27.189699 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:27.192975 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:27.197438 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:27.201283 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:27.207651 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:27.222177 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:27.247087 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:27.247924 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:27.336503 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:27.337232 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:27.337831 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:27.338114 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:27.357663 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 14:11:27.359503 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:27.364136 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:27.370045 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 14:11:27.378191 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:27.463712 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:27.464330 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:27.465415 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:27.475692 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:27.476924 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:27.481804 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:27.497423 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:27.563987 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:27.568239 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:27.570050 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:27.574728 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:27.594225 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:27.596930 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:27.599732 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:27.660759 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:27.662240 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:27.662706 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:27.672048 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:27.678031 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:27.689785 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:27.701678 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:27.749248 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:27.750319 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:27.753885 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:27.754698 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:27.757928 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:27.770568 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 14:11:27.772854 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:27.779568 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:27.797118 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:11:27.840695 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:27.874315 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:27.890954 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 14:11:27.994910 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:27.997120 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:27.999555 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:28.000504 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:28.001698 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:28.005712 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:28.006685 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:28.056154 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 14:11:28.103242 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:28.107441 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:28.113137 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:28.115077 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:28.119211 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:28.119956 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:28.141136 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:28.144819 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 14:11:28.179203 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:28.179884 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:28.191557 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:28.195199 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:28.203593 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:28.215240 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:28.226876 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:28.289418 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:28.297178 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 67 similar messages]
I20240501 14:11:28.297240 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:28.299268 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:28.299738 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:28.306339 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:28.307637 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:28.331219 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:28.343417 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:28.372378 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:28.376999 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:28.377822 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:28.384443 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:28.391254 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:28.395167 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:28.399398 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:28.496400 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:28.496882 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:28.497445 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:28.502027 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:28.511879 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:28.513085 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:28.518700 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:28.567476 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:11:28.586575 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 14:11:28.623996 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:28.625804 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:28.625921 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:28.632458 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:28.636315 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:28.639195 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:28.659575 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:28.677507 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:28.683893 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:28.684295 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:28.685632 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:28.694552 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:28.695421 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 14:11:28.708357 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:28.729889 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:11:28.736475 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:28.812108 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:28.813987 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:28.819202 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:28.823472 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:28.831229 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:28.841830 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:28.844938 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:11:28.944049 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 14:11:29.019557 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:29.020648 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:29.021811 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:29.027594 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:29.031435 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:29.051306 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:29.054198 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:29.160809 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:29.164726 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:29.166658 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:29.167598 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:29.168772 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:29.173063 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:29.188045 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:29.191660 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:29.209518 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:29.213057 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:29.228181 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:29.232854 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:29.233527 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:29.237293 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:29.240015 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:29.349295 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:29.352021 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:29.352447 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:29.356738 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:29.373342 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:29.379406 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:29.390542 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:29.391150 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:29.421546 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:29.423743 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:29.431818 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:29.434307 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:29.446753 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:29.446741 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 14:11:29.447660 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 14:11:29.449399 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:29.477670 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:29.525148 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 14:11:29.538626 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:29.541469 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:29.541566 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:29.548288 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:29.549600 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:11:29.549929 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:29.562678 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:29.571014 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:29.672278 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:29.676405 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:29.679201 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:29.690747 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:29.694959 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:29.704664 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:29.705055 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:29.786335 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:29.787413 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:29.788679 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:29.793274 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:29.793637 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:29.794261 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:11:29.815186 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 14:11:29.816881 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:29.832089 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:29.837410 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:29.916558 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:29.918453 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:29.918809 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:29.920539 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:29.921737 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:29.933421 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:29.943679 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:30.037693 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:30.037990 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:30.040484 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:30.042831 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:30.045249 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:30.048274 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:30.063474 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:30.156564 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:30.156939 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:30.164610 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:30.165892 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:30.175701 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:30.197113 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:30.199090 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:30.263243 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:30.297480 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:30.300204 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:30.301481 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:30.304749 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:30.306627 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:30.320922 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:30.322196 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:30.323570 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:30.325548 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:30.328022 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:30.333789 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:30.344141 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:30.370278 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:30.372918 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:30.379204 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:11:30.380064 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:30.390130 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:30.420231 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:30.438221 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 14:11:30.468676 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:30.475586 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:30.482070 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:30.483880 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:30.489145 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:30.493526 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:30.517210 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:30.532325 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:30.535411 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:30.541599 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:30.541956 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:30.541990 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:30.550164 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:30.558066 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:30.627317 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:30.650002 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 14:11:30.659763 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:30.661324 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:30.665338 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:30.665997 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:30.669696 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:30.684253 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:30.690285 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:30.701171 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:30.846904 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:30.847415 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:30.853312 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:30.856007 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:30.856622 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:30.872735 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:30.879415 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:30.923779 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 14:11:30.972887 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:30.974519 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:30.981197 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:30.985692 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:30.992355 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:30.997797 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:31.018514 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:31.055213 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:31.059166 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:31.062413 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:31.063988 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:31.068070 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:31.078912 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:31.096444 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:31.104534 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:11:31.167282 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:11:31.182153 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:31.185999 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:31.188373 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:31.199532 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:31.209700 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:31.212517 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:31.215220 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:31.314096 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:31.314155 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:31.316098 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:31.324718 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:31.325258 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:31.332154 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:31.361889 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:31.378880 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:31.379204 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:31.382743 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:31.388183 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:31.395896 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:31.397991 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 14:11:31.412775 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:31.423300 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:31.509202 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:31.512810 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:31.514861 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:31.515468 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:31.530301 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:31.534441 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:31.550179 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:31.578158 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:31.581425 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:31.595207 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:31.597261 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:31.597535 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:31.597771 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:31.616401 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 14:11:31.617151 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:31.617753 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:31.693529 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:31.694298 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:31.694964 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:31.705346 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:31.707356 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:31.723176 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:31.745014 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:31.837184 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 14:11:31.916268 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:31.918304 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:31.919461 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:31.921139 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:31.921139 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:31.948688 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:31.949419 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:32.024180 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:32.025697 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:11:32.029917 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 14:11:32.069347 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:32.073485 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:32.074942 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:32.077594 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:32.089457 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:32.096360 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:32.096984 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:32.106698 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:32.116560 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:32.119838 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:32.129413 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:32.143947 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:32.147336 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:32.163965 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:32.261627 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:32.263468 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:32.264598 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:32.266186 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:32.268261 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:32.268810 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:32.286551 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:32.375392 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:32.387770 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:32.388752 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:32.388756 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:32.389288 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:32.389487 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:32.390223 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:11:32.414234 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:32.492693 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 14:11:32.506896 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:32.508109 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:32.508975 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:32.509339 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:32.518702 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:32.528367 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:32.533411 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:32.539932 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:32.550093 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:32.566452 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:32.573635 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:32.575219 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:32.584051 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:32.591389 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:32.592602 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:32.702023 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:32.707785 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:32.709722 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:32.711707 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:32.715411 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:32.722386 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:32.725138 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:32.727945 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:32.728080 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:32.732460 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:32.747432 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:32.751703 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:32.753963 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:32.788144 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 14:11:32.788905 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:32.796473 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:32.862090 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:33.001591 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:11:33.020197 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 14:11:33.092368 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:33.101642 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:33.103544 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:33.105165 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:33.110855 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:33.111096 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:33.144197 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:33.150872 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:33.153926 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:33.156702 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:33.159081 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:33.159379 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:33.160065 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:33.160894 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:11:33.166304 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 14:11:33.278846 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:33.284250 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:33.285092 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:33.285106 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:33.287230 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:33.297662 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240501 14:11:33.299420 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:33.304366 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 14:11:33.334705 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:33.336280 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:33.344514 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:33.346406 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:33.350553 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:33.361187 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:33.369701 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:33.372073 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:33.450991 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:33.468432 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:33.475205 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:33.477985 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:33.478070 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:33.479457 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:33.480253 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:11:33.497975 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 14:11:33.498657 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:33.589380 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:33.593784 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:33.595002 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:33.596033 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:33.609261 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:33.623773 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:33.625677 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:33.720465 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:33.736294 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:33.743235 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:33.746083 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:33.746541 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:33.754827 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:33.759977 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:33.762370 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:33.773564 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:33.787652 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:33.788709 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:33.791268 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:33.809252 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:33.810194 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:33.824255 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 14:11:33.911761 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:33.912387 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:33.918617 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:33.926990 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:33.927348 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:33.931922 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:33.934077 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:33.938948 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:11:33.940698 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:34.123435 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:34.124428 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:34.124951 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:34.128100 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:34.139348 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:34.140086 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:34.159881 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:34.267345 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:34.268175 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:34.269577 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:34.276891 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:34.280336 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:34.288486 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:34.295485 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:34.295732 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:34.297022 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:34.297612 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:34.303268 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 14:11:34.315164 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:34.326282 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:34.326560 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:34.329749 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:34.371680 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:34.376224 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:11:34.433631 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 14:11:34.457829 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:34.470908 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:34.474431 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:34.475591 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:34.477634 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:34.478349 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:34.478978 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:34.479312 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:34.480832 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:34.491070 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:34.496325 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:34.502866 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:34.533226 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:34.547701 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:34.564083 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 14:11:34.630365 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:34.637128 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:34.637588 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:34.638224 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:34.639401 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:34.658661 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:34.675009 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:34.736974 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:11:34.754374 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:34.757565 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:34.759603 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:34.766755 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:34.767237 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:34.780532 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:34.795374 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:34.801076 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 14:11:34.871853 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:34.872321 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:34.873247 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:34.876663 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:34.876999 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:34.883909 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:34.886777 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:34.908365 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:34.943466 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 14:11:34.991575 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:34.991956 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:34.995070 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:34.999971 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:35.007926 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:35.016438 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:35.021677 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:35.104573 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:35.104574 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:35.106024 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:35.108062 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:35.109712 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:11:35.121536 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 14:11:35.124157 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:35.131405 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:35.225556 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:35.229898 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:35.231170 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:35.233827 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:35.249087 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:35.251639 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:35.253516 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 14:11:35.254312 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:35.349844 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:35.352219 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:35.367918 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:35.370859 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:35.374992 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:35.388902 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:35.391923 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:11:35.403873 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:11:35.412317 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:35.413023 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:35.416864 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:35.419697 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:35.424443 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:35.425670 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:35.428306 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:35.428836 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:35.448122 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:35.522925 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 14:11:35.530229 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:35.531826 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:35.533082 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:35.534067 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:35.535923 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:35.539563 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:35.542702 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:35.547940 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 14:11:35.602234 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:35.603224 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:35.605918 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:35.609861 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:35.614351 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:35.629880 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:35.630152 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:35.651742 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 14:11:35.719517 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:35.720448 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:35.720901 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:35.722229 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:35.724606 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:35.729080 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:35.737722 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:35.803531 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:35.853669 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:35.856602 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:35.863169 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:35.864914 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:35.867594 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:35.883636 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:35.890089 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:35.953917 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 14:11:35.977648 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:35.977916 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:35.981117 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:35.983850 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:35.984735 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:36.000093 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:36.026063 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:36.077680 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:36.080848 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:36.081470 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:36.083797 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:36.087702 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:36.109500 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:36.111961 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:36.201191 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:36.203733 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:36.212402 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:36.213706 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:36.217857 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:36.220044 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:36.226624 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:36.230269 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:36.282891 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:36.336339 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:11:36.336622 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:36.340401 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:36.349524 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:36.355489 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:36.356303 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 14:11:36.363246 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:36.365499 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:36.366102 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:36.372917 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:36.375077 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:36.380410 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:36.389858 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:36.393895 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:36.411000 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:36.426631 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:36.534289 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:36.537313 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:36.538055 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:36.544173 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:36.544524 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:36.554924 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:36.556393 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:36.597357 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:36.602818 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:36.603207 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:36.603423 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:36.609982 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:36.611099 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:36.615294 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:36.717283 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:36.718051 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:36.719322 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:36.724025 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:36.725930 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:36.742578 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:36.748601 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:36.751222 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:11:36.759544 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:36.872670 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 14:11:36.892742 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:36.894137 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:36.896030 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:36.898367 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:36.899161 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:36.912340 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:36.938360 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:37.022131 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 14:11:37.051146 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:37.060606 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:37.061096 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:37.079437 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:37.080425 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:37.081023 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:37.089385 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:37.107618 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:37.127596 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:37.129491 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:37.129936 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:37.132597 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:37.145532 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:37.153275 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:37.159381 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:37.256868 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:37.258796 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:37.258952 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:37.267459 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:37.273566 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:37.278328 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:37.280325 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:37.322798 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:37.362742 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 14:11:37.378048 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:37.383242 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:37.390565 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:37.392076 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:37.400550 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:37.403723 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:37.404098 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:37.466118 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:37.466411 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:37.471331 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:37.473544 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:37.495126 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:37.495749 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:37.497227 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:37.559290 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:37.561167 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:37.571466 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:37.572654 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:37.580415 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:37.592275 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:37.606658 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:37.639434 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:37.640226 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:37.649633 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:37.658581 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:37.661998 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:37.672686 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:37.684085 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 14:11:37.696915 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:37.712723 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:11:37.715375 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 14:11:37.764209 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:37.764990 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:37.769716 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:37.769812 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:37.770478 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:37.788512 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 14:11:37.797349 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:37.801384 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:37.839421 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:37.895517 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 14:11:37.984663 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:37.988744 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:37.990430 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:37.992172 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:37.994381 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:37.998431 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:38.005409 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:38.105576 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:38.107189 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:38.115814 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:38.119695 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:38.120003 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:38.120271 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:38.127691 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:38.135839 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:38.151886 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:38.161552 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:38.163477 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:38.174633 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:38.175068 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:38.191617 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:38.202569 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 14:11:38.291869 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:38.297881 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:38.300197 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:38.300848 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:38.308686 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240501 14:11:38.309525 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:38.310149 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:38.315723 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:38.335767 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:38.413602 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:38.415794 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 14:11:38.416122 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:38.416783 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:38.424357 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:38.424582 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:38.428226 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:38.428814 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:38.515911 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:38.516093 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:38.517390 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:38.520359 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:38.526433 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:38.533600 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:38.533946 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:38.605454 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:38.610143 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:38.611491 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:38.613142 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:38.619297 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:38.620481 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:38.628825 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:38.705977 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:38.710034 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:38.710602 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:38.719199 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:38.729920 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:38.730648 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:38.751618 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:11:38.762321 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 14:11:38.787138 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:38.789312 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:38.791368 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:38.798313 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:38.800343 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:38.800922 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:38.813700 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:38.817318 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:38.882802 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 14:11:39.045897 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:39.047638 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:39.053459 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:39.055868 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:39.067346 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:39.070622 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:39.073303 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:39.161768 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:39.172586 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:39.173331 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:39.173627 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:39.174837 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:39.195096 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:39.202649 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:11:39.211838 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 14:11:39.237478 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:39.246168 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:39.247150 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:39.247853 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:39.255944 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:39.267412 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:39.276594 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:39.332250 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 14:11:39.357543 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:39.363677 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:39.365679 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:39.366581 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:39.377732 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:39.391827 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:39.393200 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:39.397199 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:39.480221 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:39.484819 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:39.489320 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:39.490763 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:39.493063 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:39.496222 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:39.511920 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:39.536556 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 14:11:39.596560 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:39.598047 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:39.601411 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:39.603077 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:39.609287 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:39.610153 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:39.632084 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:39.691008 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:39.698906 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:39.698959 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:39.699007 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:39.711908 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:39.724516 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:39.734998 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:11:39.750744 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 14:11:39.790422 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:39.794953 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:39.799764 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:39.801986 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:39.804425 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:39.818260 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:39.829712 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:39.830578 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 14:11:39.879959 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:39.880646 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:39.883339 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:39.886946 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:39.905076 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:39.905184 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:39.911655 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:39.957938 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 14:11:39.980847 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:39.983875 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:11:40.030243 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:40.088068 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:11:40.123791 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:40.127784 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:40.130096 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:40.137878 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:40.138372 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:40.154130 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:40.165832 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:40.216923 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:40.239701 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 14:11:40.246373 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:40.248315 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:40.257880 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:40.258265 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:40.258620 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:40.285892 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:40.297362 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:40.313263 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:40.317323 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:40.318404 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:40.319845 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:40.321574 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:40.329136 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:40.340348 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:11:40.346482 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:40.365877 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:40.440033 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:40.440800 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:40.446511 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:40.447634 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:40.457067 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:40.460978 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 14:11:40.463555 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:40.465937 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:40.553575 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:40.559623 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:40.560144 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:40.570438 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:40.571245 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:40.585479 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:40.592707 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:40.623441 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 14:11:40.674129 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:40.676717 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:40.677888 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:40.689177 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:40.700012 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:40.700260 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:11:40.703951 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:40.711568 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:40.761118 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:40.762480 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:40.763509 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:40.765488 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:40.766839 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:40.770340 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:40.780975 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:40.835767 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:11:40.851573 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 14:11:40.851907 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:40.856781 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:40.862793 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:40.863564 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:40.878132 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:40.887466 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:40.905054 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:40.927356 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:40.928009 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:40.942817 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:40.949919 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:40.952085 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:40.960651 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:40.961103 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:41.223246 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:41.225924 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:41.228163 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:41.228387 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:41.229911 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:41.232369 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:41.258199 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:41.319883 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:41.321499 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:41.322688 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:41.332667 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:41.332715 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:41.333442 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:41.334091 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:11:41.349946 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:11:41.371377 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:11:41.407958 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:41.413529 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:41.416812 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:41.423293 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:41.424124 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:41.425475 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:41.432201 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:41.496017 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:41.496798 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:41.499078 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:41.499171 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 14:11:41.507153 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:41.510560 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:41.510607 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:41.516491 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:11:41.604482 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 14:11:41.610512 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:41.611011 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:41.612092 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:41.625705 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:41.626489 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:41.627800 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:41.641995 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:41.761289 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:41.762081 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 14:11:41.771605 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:41.779681 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:41.780081 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:41.784677 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:41.791117 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:41.791193 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:41.791676 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:41.798610 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:41.798892 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:41.798975 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:41.802495 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:41.819758 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:41.866077 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:41.871484 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:11:41.920701 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 14:11:41.953024 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:41.953681 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:41.962545 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:41.968349 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:41.980011 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:41.980300 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:41.980300 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:41.982342 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:41.982684 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:41.990617 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:42.000128 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:42.023748 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:42.024152 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:42.024217 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:42.068403 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:42.297426 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 14:11:42.344305 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:42.346777 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:42.347528 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:42.351567 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:42.379576 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:42.389313 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:42.397003 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:42.399708 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:42.403219 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:42.405282 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 14:11:42.410519 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:42.411824 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:42.420003 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:42.422650 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:42.428699 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:42.435832 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:42.535930 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:42.536129 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:42.542618 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 14:11:42.547087 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:42.551667 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:42.552742 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:42.565702 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:42.578115 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:42.585474 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:42.585597 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:42.598548 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 14:11:42.613209 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:42.616423 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:42.626833 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:42.629179 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:42.631232 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:42.657930 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 14:11:42.722963 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:42.724175 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:42.728453 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:42.729957 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:42.730934 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:42.753834 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:42.754033 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 14:11:42.757833 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:42.859187 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:42.860003 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:42.865048 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:42.865991 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:42.875417 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:42.879168 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:42.881577 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 14:11:42.890293 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:42.896114 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:42.945789 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:42.946146 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:42.946425 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:42.949280 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:42.959901 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:42.964145 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:42.985920 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:43.041383 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:43.043008 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:43.050827 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:43.059258 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:43.059290 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:43.062182 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:43.071157 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:43.127187 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:43.129266 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:43.130621 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:43.133740 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:43.139968 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:43.146975 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:43.163239 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:11:43.246534 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:43.318166 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:43.325193 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 68 similar messages]
I20240501 14:11:43.382447 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:43.385917 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:43.387243 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:43.391603 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:43.398532 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:43.412783 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:43.412922 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:43.414649 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:43.491427 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:43.497634 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:43.498651 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:43.509073 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:43.509474 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:43.515393 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:43.523114 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:43.574880 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:43.575330 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:43.581684 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:43.584347 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:43.589982 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:43.618283 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:43.618549 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:43.679270 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:43.680603 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:43.684041 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:43.685557 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:43.688858 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:43.697417 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:43.698745 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:43.769397 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:43.771021 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:43.772085 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:43.776346 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:43.780022 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:43.787207 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:43.791267 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:43.885937 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:43.887423 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:43.888190 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:43.888612 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:43.894527 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:43.895025 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:43.899819 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:43.900615 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:44.014604 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:44.017027 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:44.017306 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 14:11:44.021853 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:44.025998 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:44.031906 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:44.050463 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:44.061034 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:44.062750 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:44.067574 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:44.067904 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:44.069962 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:44.071646 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:44.095212 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:44.118907 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:44.202760 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 14:11:44.208693 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:44.209288 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:44.209612 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:44.210256 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:44.227229 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:44.227939 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:44.232517 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:44.313306 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:44.327013 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 14:11:44.394454 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:44.396369 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:44.401418 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:44.403168 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:44.404444 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:44.409752 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:44.427981 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:44.428050 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:11:44.429265 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:44.537815 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:44.541054 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:44.552222 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:44.553220 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:44.557726 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:44.573940 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:44.574116 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:44.576074 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:44.578431 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:44.582108 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:44.585589 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:44.588951 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:44.603430 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 14:11:44.607228 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:44.613301 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:44.738656 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:44.746073 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:44.748803 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:44.751838 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:44.759760 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:44.766826 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:44.769826 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:44.776453 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:44.786501 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:44.789207 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:44.793185 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:44.794251 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:44.799139 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:44.809223 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:44.847226 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 14:11:44.925752 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:44.926517 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:44.926651 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:44.930104 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:44.930188 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:44.939855 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:44.964941 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 14:11:44.967586 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:45.045235 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:45.046965 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:45.047492 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:11:45.050521 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 14:11:45.052973 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:45.053593 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:45.061151 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:45.063225 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:45.091467 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:45.104274 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:45.130738 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 14:11:45.142887 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:45.143108 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 14:11:45.145423 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:45.146720 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:45.156842 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:45.160557 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:45.170958 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:45.171383 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:45.236994 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:45.257997 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:11:45.269827 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:45.270823 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:45.270932 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:45.283059 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:45.283489 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:45.285077 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:45.285961 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:45.382817 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:45.384336 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:45.391571 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:45.399694 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:45.402545 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:45.407371 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:45.411986 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:45.505635 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:45.505873 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:45.510105 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:45.516726 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:45.519876 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:45.529356 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:45.550789 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:45.615447 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:45.616817 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:45.627506 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:45.631511 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:45.647387 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:45.661172 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:45.664171 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:45.676937 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:45.681619 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:45.686920 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:45.692061 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:45.695261 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:45.703914 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:11:45.708526 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:45.716014 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:45.717272 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:45.729230 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:45.801040 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 14:11:45.852670 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:45.862623 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:45.866915 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:45.870503 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:45.871150 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:45.879871 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:45.886853 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:45.889232 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:45.898842 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:45.916268 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:45.916989 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 14:11:45.924393 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:45.929075 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:45.935606 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:45.939886 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:46.022215 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:46.025691 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:46.028642 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:46.029382 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:46.032063 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:46.043505 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:46.051466 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:11:46.168709 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:46.246963 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:46.247713 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:46.252570 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:46.255014 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:46.262073 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:46.269133 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:46.281061 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:46.369982 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:46.370548 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:46.374379 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:46.379319 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:46.380196 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:46.383436 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:46.406821 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:46.443703 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:46.447896 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:46.452296 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:46.456094 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:46.465221 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:46.471107 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:46.477610 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:46.532541 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 14:11:46.557941 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:46.563724 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:46.565574 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:46.570561 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:46.581833 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:46.595124 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:46.598310 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:46.632635 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:46.663666 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:11:46.666076 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:11:46.666775 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 14:11:46.683411 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:46.685878 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:46.685927 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:46.694738 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:46.696808 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:46.707253 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:46.708355 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:46.791115 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:46.791710 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:46.793998 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:46.799216 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:46.805341 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:46.815351 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:46.817095 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:46.872396 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:46.875036 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:46.879230 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:46.879369 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:46.887251 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 14:11:46.888338 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:46.891907 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:46.915195 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:46.980171 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:46.982014 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:46.982969 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:46.984236 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:46.998459 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:47.005086 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:47.017390 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:47.060397 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:47.064297 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:47.066318 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:47.069177 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:47.083976 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:47.086735 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:47.086901 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:47.104441 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:47.190659 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:11:47.310779 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 14:11:47.311365 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:47.311573 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:47.316594 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:47.319130 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:47.323954 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:47.325543 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:47.326550 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:47.387674 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:47.444521 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 14:11:47.450459 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:47.452131 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:47.456112 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:47.459177 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:47.470199 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:47.482213 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:47.483652 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:47.484701 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:47.486294 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:47.491214 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:47.491987 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:47.503711 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:47.506283 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 14:11:47.520206 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:47.522933 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:47.621861 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:47.623567 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:47.624469 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:47.625067 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:47.626624 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:47.634687 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:47.646548 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 14:11:47.647292 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:47.666250 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 14:11:47.732790 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:47.739888 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:47.746147 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:47.747551 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:47.752135 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:47.755851 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:47.761996 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:47.774755 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:47.853338 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:47.855309 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:47.855994 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:47.856966 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:47.867676 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:47.885426 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:47.885751 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:47.938033 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:47.939267 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:47.951143 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:47.963781 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:47.965479 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:47.982367 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:47.989773 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:48.038710 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:48.044031 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:48.047706 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:48.047978 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 14:11:48.055219 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:48.059278 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:48.063027 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:48.085580 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:48.127591 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:48.131716 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:48.135231 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:48.138175 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:48.145076 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:48.147028 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:48.160809 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:48.352459 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:11:48.378592 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240501 14:11:48.382722 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:48.383730 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:48.385322 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:48.391288 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:11:48.393731 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 14:11:48.397775 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:48.399236 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:48.400408 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:48.433617 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 14:11:48.508997 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:48.509029 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:48.515699 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:48.517011 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:48.521075 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:48.526414 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:48.539029 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:48.539366 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:48.542093 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:48.553557 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:48.565387 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:48.571547 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:48.571806 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:48.577116 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:48.685879 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:48.686728 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:48.687067 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:48.687839 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:48.701354 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:48.708537 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:48.718524 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:48.740098 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:48.799693 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:48.803890 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:48.809794 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:48.814291 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:48.822121 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:48.827569 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:48.830891 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:48.946746 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:48.955360 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:48.955526 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:48.957720 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:48.965979 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:48.966998 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:48.970808 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:48.971856 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:48.973379 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:48.979656 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:48.991479 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:48.993539 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:49.003412 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:49.029214 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:49.048851 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 14:11:49.134788 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:49.138764 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:49.139595 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:49.140826 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:49.142040 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:49.144572 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:49.147332 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:49.151078 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:49.154901 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:49.155579 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:49.170662 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:49.188423 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 14:11:49.192255 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:49.210724 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:49.216917 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:49.228049 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:11:49.270648 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:49.328699 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:49.450475 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 14:11:49.531407 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:49.532851 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:49.535096 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:49.542467 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:49.547205 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:49.553973 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:49.559229 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:49.562822 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:49.563527 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:49.582402 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:49.590811 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:49.599395 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:49.604128 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 14:11:49.606112 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:49.617569 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:49.719393 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:49.724524 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:49.724712 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:49.752872 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:49.753768 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:49.754487 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:49.754696 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:49.763193 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:49.770961 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:49.792872 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:49.794778 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:49.795917 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:49.796067 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:49.805763 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:49.821671 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:11:49.873798 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 14:11:49.900374 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:49.901835 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:49.903717 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:49.908511 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:49.916172 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:49.918829 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:49.931906 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 14:11:49.936410 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:49.988847 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 14:11:50.023612 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:50.024547 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:50.027137 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:50.032501 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:50.033166 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:50.041028 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:50.043430 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:50.143065 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:50.144555 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:50.145543 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:50.150435 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:50.157900 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:50.159893 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:50.179337 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:50.199155 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:50.208065 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 14:11:50.224984 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:50.225661 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:50.226426 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:50.228586 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:50.230082 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:50.241886 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:50.242926 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:50.258813 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 14:11:50.323437 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:50.324100 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:50.326211 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:50.326920 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:50.334298 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:50.340008 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:50.342339 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:50.342527 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:50.396297 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:11:50.400949 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:50.496145 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:11:50.537392 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:50.537808 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:50.539031 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:50.540560 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:50.543361 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:50.551420 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:50.585351 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:50.663271 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:50.667855 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:50.668447 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:50.674259 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:50.681371 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:50.682521 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:50.687079 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:50.747468 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:50.751240 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:50.765770 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:50.767105 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:50.768498 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:50.772142 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:50.779502 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:50.820273 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:11:50.843673 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:50.850134 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:50.854856 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:50.855479 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:50.863862 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:50.864152 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:50.866761 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 14:11:50.868741 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:50.939563 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:50.940837 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:50.944536 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:50.944697 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:50.954775 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:50.961948 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:50.966624 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:50.976338 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:51.059583 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:51.062136 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:51.064289 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:51.076077 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:51.078708 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:51.086012 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:51.095715 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:51.103772 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 14:11:51.199204 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:51.199417 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:51.204023 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:51.207264 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:51.208492 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:51.220929 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:51.221269 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:51.261233 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:51.261715 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:51.263321 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:51.269814 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:51.271171 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:51.280202 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:51.283151 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:51.380713 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:51.381704 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:51.384199 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:51.390209 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:51.402750 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:51.407195 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:51.417644 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:11:51.467818 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:51.545444 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:51.545472 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:51.548439 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:51.551501 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:51.559844 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:51.573804 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:51.579435 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:51.673792 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:51.674196 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:51.675511 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:51.679345 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:51.685946 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:51.691754 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 14:11:51.692524 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 14:11:51.695951 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:51.705430 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:51.752252 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:51.772086 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 14:11:51.791004 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:51.791396 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:51.792531 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:51.797509 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:51.798264 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:51.823797 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:51.824919 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:51.867071 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:51.869104 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:51.875130 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:51.875739 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:51.892290 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:51.897773 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:51.901504 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:51.990747 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:51.993196 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:51.995676 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:51.998550 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:52.004148 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:52.008324 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:52.008832 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 14:11:52.025224 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:52.070101 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:52.075573 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:52.076015 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:52.076970 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:52.077100 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:52.077203 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:52.082213 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:52.087918 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:52.212097 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:52.212880 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:52.218713 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:52.221300 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:52.228477 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:52.231112 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:52.248440 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 14:11:52.256829 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:52.318550 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 14:11:52.327941 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:52.335763 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:52.336455 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:52.338016 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:52.338447 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:52.339763 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:52.353672 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:52.380069 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 14:11:52.444269 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:52.444572 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:52.446611 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:52.447016 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:52.459728 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:52.466431 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:52.467499 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:52.475917 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:52.546413 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:52.551019 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:52.554594 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:52.559875 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:52.563482 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:52.567219 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:52.586266 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:52.652823 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 14:11:52.672159 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:52.673449 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:52.677615 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:52.680428 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:52.689356 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 14:11:52.690789 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:52.699388 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:52.699807 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:52.725658 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:52.758651 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 14:11:52.795352 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:52.797113 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:52.799124 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:52.806380 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:52.831125 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:52.836319 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:52.849103 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:52.857158 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:52.865114 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:52.867539 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:52.867691 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:52.876965 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:52.894729 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:52.895576 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:52.902180 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:11:52.956440 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 14:11:52.998880 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:52.999783 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:52.999821 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:53.000908 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:53.012182 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:53.030092 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:53.030822 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:53.031211 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:53.041829 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:53.042963 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:53.058691 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:53.062331 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:53.067118 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:53.074849 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:53.170251 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:53.170747 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:53.177932 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:53.191704 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:53.193264 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:53.201299 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:53.202884 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 14:11:53.209259 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:53.379971 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:53.380940 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 68 similar messages]
I20240501 14:11:53.387149 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:53.389604 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:53.391722 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:53.393246 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:53.397933 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:53.399863 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:53.408007 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:53.500741 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:53.510646 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:53.512342 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:53.519536 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:53.529734 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:53.535879 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:53.546123 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:53.551829 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:53.561373 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:53.568337 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:53.571961 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:53.580297 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:53.582082 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:53.592525 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:53.681190 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 14:11:53.690135 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:53.693814 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:53.698563 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:53.701676 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:53.705189 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:53.706753 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:53.714649 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:53.805537 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 14:11:53.816674 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:53.821010 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:53.821163 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:53.823961 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:53.824472 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:53.827524 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:53.856850 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:53.863675 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:53.924597 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:53.928050 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:53.929947 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:53.931628 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:53.938316 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:53.938324 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:53.955796 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:54.025255 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:54.030143 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:54.032150 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:54.034477 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:54.039445 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:54.041059 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:54.053660 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:54.088953 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:54.097589 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:54.107884 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:54.124536 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:54.136157 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:54.143679 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:54.143764 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:54.236693 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:54.247202 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:54.247465 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:54.257822 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:54.261270 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 14:11:54.265327 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:54.266203 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:54.271719 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:54.279016 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:54.401875 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:54.462195 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 14:11:54.465370 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 14:11:54.473078 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:54.474167 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:54.476438 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:54.478536 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:54.479691 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:54.488952 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:54.524746 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:54.648561 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:54.653811 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:54.660962 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:54.660962 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:54.671921 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:54.672399 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:54.676394 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:54.686856 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:54.688566 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:54.700021 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:54.701331 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:54.716529 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:54.725179 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:54.725698 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:54.746143 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:54.810839 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:54.832720 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 14:11:54.840188 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:54.840531 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:54.841856 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:54.842265 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:54.847008 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:54.853636 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:54.861274 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:54.955334 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:54.961153 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:54.961997 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:54.964093 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:54.965498 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:54.978535 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:54.996204 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:55.027137 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 14:11:55.070399 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:55.071848 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:55.077004 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:55.080204 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:55.084952 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:55.091890 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:55.093914 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:55.141803 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:55.149150 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:55.150182 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 14:11:55.150739 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:55.156289 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:55.156771 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:55.156929 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:55.163167 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:55.163718 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:55.180817 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:55.202095 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:55.223372 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 14:11:55.261207 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:55.267261 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:55.279227 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:55.281666 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:55.291539 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:55.306003 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:55.307402 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:55.311569 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:55.311759 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:55.317471 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:55.322005 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:55.323243 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:55.323683 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:55.348690 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:55.398335 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:55.487160 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 14:11:55.487766 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 14:11:55.655273 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:55.662333 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:55.662770 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:55.672534 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:11:55.677723 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:55.681540 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:55.681813 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:55.698954 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:55.699301 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:55.706912 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:55.712188 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:55.716775 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:55.717249 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:55.722846 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:55.762181 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:55.768028 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:55.849997 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:55.857836 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:55.858842 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:11:55.865041 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:55.867421 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:55.869803 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:55.882879 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:55.896893 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:55.920972 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:55.925053 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:55.925365 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:55.936013 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:55.936746 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:55.952545 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:55.953042 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:56.047019 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:56.047765 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:56.047807 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:56.048804 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:56.050596 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:56.068523 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:56.071333 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:56.113408 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:56.141068 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:56.164515 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:56.166846 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:56.168442 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:56.172484 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:56.181221 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:56.182801 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:56.195855 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:56.277585 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:56.278435 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:56.286916 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:56.293663 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:56.299191 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:56.299505 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:56.309978 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 14:11:56.330889 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:56.339872 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:56.341051 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:56.345607 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:56.349736 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:56.352291 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:56.356612 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:56.375304 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:56.460522 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:56.461333 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:56.466250 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:56.468035 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:56.472250 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:56.490989 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:56.492609 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:56.566565 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:56.678754 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:56.679651 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:56.683950 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:56.689924 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:56.697480 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:56.709405 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:56.712278 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:56.746277 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 14:11:56.793102 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:56.797178 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:56.803879 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:56.806901 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:56.809404 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:56.810070 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:56.820223 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:56.875952 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:56.879755 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:56.879925 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:56.880230 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:56.880722 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:56.881982 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:11:56.912534 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:11:56.921512 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:56.982510 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:56.984194 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:56.987597 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:56.988498 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:57.001391 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:57.015869 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 14:11:57.030660 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:57.032815 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:57.040699 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:57.052729 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:57.055713 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:57.056106 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:57.059944 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:57.068311 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:57.079000 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:57.079459 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:11:57.164077 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 14:11:57.176060 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:57.178490 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:57.178958 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:57.182817 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:57.184005 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:57.196295 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:57.214329 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:57.292799 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:57.293047 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:57.300278 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:57.303882 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:57.304489 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:57.307351 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:57.308091 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:57.340467 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:57.349076 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 14:11:57.401603 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:57.402375 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:57.405478 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:57.409902 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:57.415985 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:57.428717 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:57.432236 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:57.478889 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 14:11:57.510144 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:57.511404 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:57.512367 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:57.512888 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:57.513491 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:57.533717 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:57.538811 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:57.562682 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:11:57.562973 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 14:11:57.622334 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:57.623085 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:57.630502 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:57.633177 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:57.641105 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:57.643678 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:57.664856 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 14:11:57.667066 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:57.700196 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:57.755878 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 14:11:57.758105 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:57.762733 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:57.763242 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:57.763741 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:57.764484 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:57.764811 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:57.779368 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:57.822139 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 14:11:57.876919 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:57.887708 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:57.889146 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:57.889219 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:57.892441 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:57.901859 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:57.904830 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:57.942637 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:57.942806 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:57.944536 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:57.945168 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:57.948773 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:57.971235 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:57.974869 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:57.998513 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:58.005255 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:58.043673 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:58.044875 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:58.050662 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:58.050861 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:58.059634 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:58.060726 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:58.065613 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:58.124130 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:58.129170 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:58.132100 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:58.134594 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:58.134631 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:58.141748 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:58.152614 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:58.243861 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:58.244112 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:58.256542 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:58.257669 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:58.268537 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:58.269605 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:58.279749 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:58.361091 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:58.365339 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:58.373371 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:58.375619 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:58.383963 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:58.390027 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:58.396167 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:58.407820 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:11:58.427280 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240501 14:11:58.479290 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:58.481137 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:58.487214 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:58.488850 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:58.490018 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:58.502707 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:58.508558 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:58.594188 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:58.595402 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:58.595952 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:58.599210 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:58.607406 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:58.620374 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:58.632237 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:58.718056 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:58.719058 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:58.719455 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:58.721390 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:58.728688 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:58.733090 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:58.760296 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 14:11:58.767084 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:58.830310 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:58.836017 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:58.838605 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:58.846592 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:58.871176 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:58.872668 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:58.879334 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:58.890311 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:58.903434 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:58.911312 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:58.917029 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:58.924047 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:58.928723 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:58.930575 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:58.947641 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:59.028877 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:59.032194 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:59.034904 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 14:11:59.035765 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:59.039258 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:59.042829 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:59.051219 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:59.067202 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:59.095927 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:59.106289 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:59.107542 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:59.107860 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:59.109678 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:59.111848 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:59.115510 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:59.224179 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:59.224517 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:59.226119 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:59.226214 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:59.226265 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:59.239322 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:59.243470 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:11:59.244262 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:59.270172 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 14:11:59.368203 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:59.370868 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:59.371009 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:59.371865 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:59.372915 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:59.385996 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:59.404376 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:59.436096 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:11:59.481889 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:59.485309 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:59.486496 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:59.494437 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:59.495219 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:59.511176 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:59.513705 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:11:59.529799 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:11:59.530545 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 14:11:59.593964 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:59.598245 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:59.600126 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:59.603822 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:59.611932 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:59.624702 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:59.631217 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:59.704869 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:59.713739 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 14:11:59.714072 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:59.717026 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:59.719847 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:59.725507 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:59.727188 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:59.747485 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:59.750097 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:11:59.771239 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 14:11:59.839572 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:59.842319 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:59.855268 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:59.860428 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:59.872629 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:59.882055 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:59.883751 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:59.885780 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:59.900215 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:11:59.908005 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:59.909303 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:11:59.930243 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:11:59.935976 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:11:59.936921 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:11:59.951718 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:00.027814 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:00.030325 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:00.034018 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:00.041438 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:00.042124 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 14:12:00.053665 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:00.056418 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:00.058702 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:00.085587 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:00.098618 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:00.102569 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:00.104389 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:00.114768 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:00.120858 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 14:12:00.127331 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:00.128671 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:00.208317 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:00.213462 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:00.215567 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:00.221632 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:00.232496 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:00.233150 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:12:00.247109 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:12:00.247983 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 14:12:00.248585 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:00.358332 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:00.363099 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:00.367249 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:00.369622 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:00.375816 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:00.377897 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:00.393256 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:00.475808 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:00.477671 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:00.480981 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:00.482326 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:00.487970 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:00.501696 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:00.505805 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:00.564464 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:00.585225 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:00.586355 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:00.586499 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:00.588383 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:12:00.588409 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 14:12:00.589097 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 14:12:00.591349 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:00.613656 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:00.617933 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:00.698299 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:00.700744 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:00.701023 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:00.704129 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:00.704607 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:00.716944 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:00.722466 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:00.723181 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 14:12:00.763279 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 14:12:00.828474 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:00.829574 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:00.832247 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:00.849702 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:00.850404 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:00.853113 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:00.877694 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:00.887470 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:00.887787 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:00.891490 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:00.895102 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:00.897562 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:00.898665 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:00.906611 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:01.006474 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:01.011703 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:01.013506 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:01.016474 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:01.018374 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:01.024351 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:01.039893 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:01.048512 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:01.076465 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:01.083992 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:01.089869 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:01.090196 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:01.090963 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:01.092186 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:01.102955 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:01.197961 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:01.199012 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:01.200960 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:01.209074 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 14:12:01.210511 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:01.211354 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:01.211357 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:01.233546 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:01.252220 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:12:01.374539 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:01.377156 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:01.380378 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:01.385087 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:01.387475 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:01.399718 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:01.408026 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:01.442375 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 14:12:01.497421 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:01.500970 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:01.501422 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:01.504758 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:01.505028 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:01.511520 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:01.515794 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:01.576191 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:01.576954 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:01.580394 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:01.584486 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:01.588027 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:01.592700 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:01.594254 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:12:01.617743 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:12:01.652107 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 14:12:01.696678 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:01.697921 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:01.698552 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:01.699851 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:01.705621 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:01.727399 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:01.728763 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:01.769925 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 14:12:01.821720 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:01.824342 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:01.827551 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:01.828394 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:01.832944 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:01.839246 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:01.863517 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:01.897631 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 14:12:01.902940 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:01.906725 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:01.909757 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:01.920969 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:01.923898 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:01.927389 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:01.950639 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:02.002086 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:02.005538 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:02.011425 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:02.012169 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:02.029111 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:02.037302 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:02.037550 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:02.094573 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:02.102681 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:02.104012 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:02.106405 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:02.114645 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:02.115295 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:12:02.122345 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:02.122821 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:02.193313 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:02.194512 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:02.196209 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:02.196713 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:02.198276 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:02.204723 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 14:12:02.212817 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:02.227085 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:12:02.238466 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:02.290503 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:12:02.337574 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 14:12:02.427560 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:02.427644 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:02.430650 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:02.434026 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:02.442246 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:02.448021 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:02.451807 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:02.551213 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:02.551581 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:02.551954 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:02.565497 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:02.575410 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 14:12:02.575950 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:02.580201 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:02.583372 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:02.583700 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:02.593721 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:02.600454 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:02.602614 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:02.619477 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:02.622709 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:02.642712 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:02.645537 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:12:02.683629 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:12:02.683820 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:12:02.717310 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 14:12:02.744094 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:02.745020 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:02.745465 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:02.750497 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:02.752774 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:02.761685 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:02.761763 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:02.870152 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:02.870474 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:02.874404 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:02.876585 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:02.885746 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:02.887185 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:02.899446 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:02.981099 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 14:12:02.982311 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:02.990669 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:02.991789 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:02.992452 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:02.994824 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:03.012590 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:03.025899 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:03.052695 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:03.063877 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:03.075208 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:03.075759 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:03.083323 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:03.086468 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:03.088128 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:03.157935 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:03.168892 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:03.169312 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:03.170328 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:03.170588 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:03.171123 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:03.180316 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:03.199826 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 14:12:03.252025 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:03.253736 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:03.253791 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:03.258553 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:03.259243 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:12:03.267763 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:03.269106 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:03.281157 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:03.489861 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:03.489953 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:03.492288 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:03.492478 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:03.495079 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:03.495126 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:03.499118 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240501 14:12:03.508133 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:03.614002 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:03.616046 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:03.617421 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:03.619748 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:03.626899 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:03.631122 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:03.635157 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:03.643817 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:03.655216 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:03.655570 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:03.660678 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:03.662122 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:03.663235 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:03.669903 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:03.688784 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:03.791785 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:03.793676 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:03.802134 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:03.802999 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:03.804842 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:03.814344 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:03.825681 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:03.912197 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:03.915311 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:03.915493 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:03.917053 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:03.917418 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:03.923954 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:03.939015 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:12:03.958145 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:12:03.999126 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 14:12:04.045068 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:04.045249 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:04.054003 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:04.059953 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:04.064461 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:04.077345 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:04.077628 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:04.078313 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:04.078684 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:04.092574 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:04.095312 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:04.100093 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:04.103915 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:04.129492 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:04.134696 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:12:04.137858 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:04.236881 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:04.241441 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:04.245981 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:04.248165 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:04.248438 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:04.248777 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:04.250435 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:04.261171 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:04.262674 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:04.272397 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:04.283407 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:04.290225 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:04.294631 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:04.307667 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 14:12:04.330471 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:04.371222 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:12:04.539446 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 14:12:04.599900 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:04.610620 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:04.617480 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:04.625569 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:04.625648 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:04.631999 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:04.643215 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:04.667438 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:04.687975 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:04.689060 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:04.689479 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:04.691859 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:04.695111 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:04.696185 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:04.715621 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:04.757460 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 14:12:04.793107 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:04.798651 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:04.801615 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:04.804097 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:04.804335 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:04.812925 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:04.830117 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:04.881114 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 14:12:04.881515 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:04.882517 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:04.890143 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:04.890233 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:04.891836 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:04.892622 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 14:12:04.900600 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:04.915926 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:04.997581 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:04.998945 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:05.003432 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:05.004184 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:05.005414 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:05.009717 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:05.022439 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:05.027565 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:05.123673 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 14:12:05.139775 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:05.141834 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 14:12:05.142086 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:05.142441 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:05.145293 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:05.145334 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:05.149183 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:05.160154 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:05.160743 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:05.163205 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:05.180513 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:05.182566 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:05.191203 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:05.202226 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:05.229183 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:12:05.287665 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:12:05.312054 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 14:12:05.323506 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:05.325196 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:05.336105 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:05.345615 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:05.347721 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:05.349076 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:05.352871 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:05.354259 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:05.354753 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:05.359706 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:05.364563 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:05.365003 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:05.414232 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:05.417071 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:05.442482 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:12:05.597072 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:12:05.673483 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 14:12:05.717586 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:05.733487 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:05.734064 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:05.738261 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:05.741092 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:05.750152 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:05.759959 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:05.762539 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:05.763172 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:05.787114 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:05.792618 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:05.792851 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:05.793684 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:05.805260 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:05.880434 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 14:12:05.927145 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:05.927631 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:05.928340 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:05.934114 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:05.939424 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:05.959323 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:05.962834 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:05.986088 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:05.987370 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:05.989230 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:05.991192 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:06.007210 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:06.019410 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:06.022779 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:06.096686 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:06.098891 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:06.102716 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:06.105551 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:06.105779 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:06.107538 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:06.111981 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:06.122928 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:12:06.134138 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:12:06.221953 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:06.225000 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:06.225462 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:06.228175 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:06.229765 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:06.233150 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:06.255553 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:06.346166 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:06.354005 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:06.354513 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:06.355221 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:06.356599 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:06.359967 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:06.377346 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:06.389288 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:06.391778 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:06.392776 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:12:06.400704 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:12:06.403981 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:06.413641 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:06.413893 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:06.426316 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:06.441433 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:06.516650 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:06.517844 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:06.523250 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:06.524083 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:06.524477 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:06.533154 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:06.553748 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:06.603286 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:12:06.603827 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:12:06.659778 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 14:12:06.727528 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:06.730154 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:06.733590 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:06.734202 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:06.734202 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:06.739843 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:06.747172 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:06.813174 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 14:12:06.869936 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:06.873011 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:06.875525 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:06.880232 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:06.883203 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:06.885746 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:06.896713 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:06.903358 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:06.905800 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:06.906915 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:06.910243 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:06.928472 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 14:12:06.932482 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:06.932977 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:06.938436 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:07.057602 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:07.059541 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:07.064925 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:07.067152 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:07.068369 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:07.069619 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:07.072729 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:07.081835 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:07.082351 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:07.083940 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:07.105170 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:07.107389 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:07.110000 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:07.121258 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:07.135316 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:07.227157 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:07.232411 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:07.234691 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:07.234961 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:07.240795 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:07.244741 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:07.248833 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:07.271773 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 14:12:07.343896 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:07.343976 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:07.344507 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:07.344707 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:07.345527 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:07.348073 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:07.348371 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 14:12:07.375934 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:07.391353 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:12:07.444020 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 14:12:07.456559 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:07.456586 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:07.461280 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:07.461982 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:07.463243 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:07.464782 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:07.467891 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:07.578419 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:07.579325 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:07.583825 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:07.584633 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:07.585944 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:07.591328 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:07.592665 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:07.599627 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:07.631942 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:12:07.660549 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 14:12:07.683008 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:07.686777 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:07.691334 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:07.696002 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:07.701802 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 14:12:07.701972 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:07.704720 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:07.712550 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:07.727998 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:12:07.803807 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 14:12:07.815325 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:07.822398 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:07.825441 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:07.825829 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:07.831213 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:07.832168 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:07.871279 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:07.949985 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:07.952697 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:07.953044 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:07.955665 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:07.964915 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:07.974174 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:08.001039 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:08.015605 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:08.034777 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:08.035271 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:08.037734 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:08.037882 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:08.038247 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:08.047459 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:08.077263 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:08.144106 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:08.146342 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:08.153540 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:08.153923 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:08.159637 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:08.169016 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:08.169098 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:08.200378 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:08.207885 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:08.212126 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:08.221500 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:08.226876 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:08.230753 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:08.236945 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:08.288400 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 14:12:08.328548 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:08.329120 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:08.332026 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:08.333986 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:08.335227 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:08.335443 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:08.337770 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:08.485970 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:08.514238 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:08.517899 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:08.519130 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:08.519810 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:08.520602 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:08.532521 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:08.533226 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:08.546202 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240501 14:12:08.633924 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:08.634636 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:08.638523 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:08.647215 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:08.650796 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:08.654042 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:08.671545 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:08.713198 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:08.716300 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:08.721634 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:08.721799 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:08.728986 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:08.733369 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:08.738076 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:08.773851 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 14:12:08.836895 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:08.837909 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:08.848582 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:08.852145 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:08.855389 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:08.856819 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:08.864279 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:08.890062 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:08.977794 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:08.980103 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:08.982530 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:08.983543 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:08.994393 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:09.000944 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:09.008383 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:09.016284 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:09.016551 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:09.031540 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:09.039065 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 14:12:09.042531 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:09.043080 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:09.043763 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:09.056195 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:09.151533 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:09.162572 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:09.163650 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:09.165688 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:09.167909 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:09.180208 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:09.191696 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:09.194906 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:09.205746 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:09.208313 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:09.220494 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:09.226002 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:09.241204 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:09.248404 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:09.341935 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:09.345903 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:09.346511 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:09.347422 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:09.352627 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:09.356036 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:09.361701 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:09.443876 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:12:09.461583 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:09.546183 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:09.551086 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:09.551689 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:09.552177 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 14:12:09.554770 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:09.556211 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:09.573535 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:09.574505 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:12:09.586134 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:12:09.604274 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:12:09.605067 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 14:12:09.677052 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:09.682284 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:09.684916 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:09.686982 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:09.695574 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:09.711241 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:09.713757 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:09.750690 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:09.760623 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:09.766095 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:09.768433 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:09.768692 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:09.777325 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:09.779785 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:12:09.793838 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 14:12:09.881260 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:09.883211 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:09.883821 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:09.884495 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:09.897910 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:09.902416 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:09.915859 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:09.969573 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:10.007728 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:10.012198 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:10.015847 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:10.035601 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:10.040760 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:10.047415 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:10.048656 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:10.080111 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:10.080107 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:10.088804 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:10.091962 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:10.092260 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:10.093473 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:10.106806 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:10.126791 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:12:10.145038 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:12:10.153510 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 14:12:10.198169 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:10.205618 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:10.212086 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:10.212302 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:10.213404 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:10.216198 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:10.222837 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:10.255896 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:10.263299 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:12:10.265194 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 14:12:10.265951 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:10.266002 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:10.266400 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:10.273741 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 14:12:10.286836 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:10.304915 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:10.349503 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 14:12:10.371870 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:10.377954 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:10.377959 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:10.380867 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:10.381016 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:10.391582 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:10.407501 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:10.493153 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:10.591172 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:10.591226 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:10.594837 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:10.596589 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:10.599215 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:10.600203 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:10.604930 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:10.668812 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 14:12:10.721769 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:10.726612 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:10.727262 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:10.735423 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:10.738740 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:10.740257 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:10.749011 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:10.749126 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:10.773212 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:10.783592 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:10.784718 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:10.793797 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:10.795889 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:10.805720 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:10.901921 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:10.906695 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:10.907408 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:10.915450 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:10.921831 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:10.924901 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:10.939953 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:10.982437 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:12:11.004779 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 14:12:11.019305 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:11.024305 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:11.029300 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:11.029779 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:11.031203 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:11.032480 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:11.033761 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:11.130721 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:11.130892 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:11.132586 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:11.133805 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:11.147192 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:11.154729 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:11.158454 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:11.212761 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:11.221619 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:11.231611 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:11.231729 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:11.232070 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:11.234970 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:11.244952 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:11.290120 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 14:12:11.296779 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:11.308034 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:11.314793 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:11.317369 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:11.321079 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:11.321092 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:11.323824 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:11.401727 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:12:11.402609 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:11.402899 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 14:12:11.407595 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:11.409278 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:11.409893 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:11.420965 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:11.423547 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:11.428884 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:12:11.540321 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:11.628494 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:11.631259 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:11.634387 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:11.635870 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:11.636080 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:11.660126 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:11.661202 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:12:11.661752 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:12:11.774776 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:11.775754 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:11.780164 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:11.780421 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:11.781772 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:11.791985 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:11.792760 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:11.800027 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:11.807618 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:11.814543 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:11.822190 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:11.836318 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:11.837559 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:11.873420 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:11.895624 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 14:12:11.956733 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:11.957561 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:11.958490 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:11.959089 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:11.964435 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:11.972419 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:11.988878 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:11.994066 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 14:12:12.070219 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:12.075304 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:12.075341 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:12.075805 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:12.082006 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:12.082296 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:12.091485 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:12.116335 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:12:12.119287 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 14:12:12.179777 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:12.184540 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:12.199663 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:12.199693 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:12.210594 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 14:12:12.215399 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:12.219398 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 14:12:12.220005 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:12.221643 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:12.260955 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:12.269114 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:12.282691 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:12.283905 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:12.287770 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:12.291652 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:12.301331 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:12.419050 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:12.424322 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:12.435273 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:12.443948 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:12.445942 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:12.448730 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:12.449975 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:12.452842 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:12.459198 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:12.462344 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:12.462615 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:12.463332 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:12.484862 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 14:12:12.487351 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:12.521780 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:12.612305 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:12:12.649497 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:12:12.675469 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:12:12.700492 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:12:12.743256 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 14:12:12.799656 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:12.804813 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:12.805750 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:12.812914 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:12.815371 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:12.820231 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 14:12:12.838762 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:12.848637 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:12.859071 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:12.879578 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 14:12:12.893266 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:12.895684 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:12.898005 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:12.898406 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:12.905298 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:12.905982 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:12.986709 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:12.994593 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:13.001188 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:13.004801 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:13.015677 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:13.018415 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 14:12:13.018573 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:13.034092 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:13.073530 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:13.074008 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:13.074193 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:13.076552 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:13.078944 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:13.092491 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:13.097416 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:13.117847 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:13.195909 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:13.198679 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:13.202354 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:13.202617 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:13.203197 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:13.206656 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:13.225260 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:13.331439 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:13.333503 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:13.336889 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:13.339263 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:13.344723 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:13.347081 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:13.362558 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:13.383374 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:13.389060 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:13.404418 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:13.404492 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:13.417171 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:13.418005 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:13.418809 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:13.516414 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:13.519728 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:13.527367 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:13.527784 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:13.528098 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:13.542838 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:12:13.561244 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 72 similar messages]
I20240501 14:12:13.581108 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:13.582746 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:13.584966 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:13.593688 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:13.613204 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:13.613698 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:13.614228 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:13.617297 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:13.631304 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:12:13.740978 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:12:13.813581 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 14:12:13.900166 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:13.902441 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:13.902967 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:13.905303 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:13.911697 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:13.931231 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:13.933055 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:13.979528 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:13.980587 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:13.980966 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:13.983747 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:13.999639 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:14.000335 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:14.004518 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:14.087994 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:14.096046 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:14.096875 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:14.096937 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:14.096928 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:14.106231 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:14.116333 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:14.164116 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:14.165380 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:14.166906 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:14.171768 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:14.178030 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:14.187340 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:14.190163 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:14.230659 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:14.260285 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:14.264216 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:14.268157 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:14.273329 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:14.273876 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:14.275379 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:14.280145 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:14.351984 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 14:12:14.375675 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:14.379129 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:14.379499 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:14.400005 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:14.400774 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:14.401736 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 14:12:14.406571 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:14.409222 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 14:12:14.420827 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:14.489177 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 14:12:14.516076 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:14.516918 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:14.518885 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:14.520223 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:14.522503 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:14.530344 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:14.533406 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:14.537272 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:14.542078 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:14.547487 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:14.560739 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:14.574843 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:14.579699 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:14.597582 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:14.686739 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:14.687206 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:14.689785 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:14.693531 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:14.709584 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240501 14:12:14.712484 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:14.712888 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:14.720564 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:14.778798 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:12:14.846060 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 14:12:14.901942 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:14.902235 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:14.906937 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:14.913506 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:14.923483 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:14.927819 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:14.935210 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:15.019076 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 14:12:15.028096 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:15.031451 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:15.035470 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:15.041318 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:15.051321 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:12:15.059600 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 14:12:15.062692 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:15.064360 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:15.077536 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:15.080199 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:15.089368 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 14:12:15.089972 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:15.093955 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:15.109584 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:15.110131 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:15.113318 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:15.129549 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:15.203528 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:15.207015 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:15.219815 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:15.221756 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:15.227259 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:15.236115 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:15.247627 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:15.286556 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:15.291823 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:15.298501 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:15.298555 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:15.299062 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 14:12:15.301635 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:15.301811 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:15.303731 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:15.315645 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:12:15.382292 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 14:12:15.411237 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:15.419718 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:15.421752 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:15.422962 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:15.430763 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:15.437017 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:15.437501 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:15.538566 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:15.541471 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:15.541864 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:15.541950 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:15.545338 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:15.555598 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:15.563992 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:15.576901 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 14:12:15.625751 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:15.627188 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:15.628708 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:15.628849 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:15.639365 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:15.640914 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:15.659792 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:15.743638 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:15.752133 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:15.752763 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:15.754490 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:15.755781 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:15.756194 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:15.776026 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:15.882513 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:15.883759 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:15.885996 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:15.887631 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:15.887687 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:15.889605 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:15.907187 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:15.923858 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:16.006728 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:16.006803 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:16.007504 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:16.015736 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:16.020047 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:16.020817 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:16.022987 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:16.052184 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 14:12:16.111136 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:16.111186 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:16.113698 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:16.119779 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:16.137027 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:16.140740 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:16.149281 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:16.182852 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:16.183321 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:16.185353 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:16.185642 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:16.191953 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:16.194885 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:16.219947 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:16.286204 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:16.290642 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:16.291272 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:16.302865 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:16.312132 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:16.313647 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:16.319324 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:16.339094 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 14:12:16.371342 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:16.379843 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:16.381289 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:16.397374 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:16.398007 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:16.411013 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:16.411676 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:16.495966 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:16.496482 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:16.497773 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:16.500825 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:16.502167 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:16.520304 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:16.530056 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:16.534004 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:12:16.552402 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 14:12:16.553248 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 14:12:16.615159 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:16.616268 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:16.617101 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:16.617735 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:16.622669 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:16.628939 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:16.636449 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:16.725778 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:16.729394 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:16.729934 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:16.732050 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:16.733342 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:16.735334 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:16.736531 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:12:16.737334 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 14:12:16.758127 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:16.829349 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:12:16.843696 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:16.847515 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:16.849061 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:16.853262 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:16.856197 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:16.863173 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:16.870693 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:16.908526 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 14:12:16.964766 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:16.967339 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:16.967415 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:16.973232 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:16.981624 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:16.984632 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:17.007526 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:17.059404 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:17.094496 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:17.100718 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:17.107481 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:17.109251 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:17.112913 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:17.116428 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:17.119491 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:17.143293 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:17.147858 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:17.155220 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:17.162351 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:17.163748 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:17.163967 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:17.197546 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:17.198318 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 14:12:17.273795 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:17.274348 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:17.274528 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:17.276670 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:17.280356 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:17.288244 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:17.306810 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:17.351902 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:17.354387 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:17.355098 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:17.358122 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:17.363308 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:17.364356 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:17.384086 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:17.386269 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:12:17.392313 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 14:12:17.475925 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:17.477366 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:17.478024 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:17.484166 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:17.484220 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:17.498865 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:17.509008 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:17.550415 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 14:12:17.601089 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:17.601681 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:17.603363 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:17.609211 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:17.621610 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:17.629303 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:17.630113 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:17.661141 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:12:17.710635 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 14:12:17.716316 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:17.716714 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:17.720794 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:17.723347 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:17.737895 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 14:12:17.741127 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:17.747787 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:12:17.751873 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:12:17.757401 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:17.767257 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:17.830652 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:17.831820 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:17.839906 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:17.841959 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:17.845299 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:17.861588 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:17.875234 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:12:17.938352 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 14:12:17.971235 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:17.974282 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:17.979673 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:17.981213 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:17.982210 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:17.992174 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:17.999861 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:18.095479 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:18.099169 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:18.104703 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:18.113013 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:18.115629 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:18.115787 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:18.133476 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:18.151242 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:18.168031 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:18.168252 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:18.168340 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:18.179735 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:18.182299 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:18.215607 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:18.259548 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:18.259600 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:18.266074 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:18.266556 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:18.267177 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:18.267416 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:18.268595 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:18.273240 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:12:18.296808 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 14:12:18.361761 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:18.362543 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:18.364038 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:18.365016 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:18.365025 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:18.370275 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:18.374092 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:18.477380 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:18.478024 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:18.479590 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:18.480072 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:18.480523 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:18.481760 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:18.494081 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:18.590790 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:18.592075 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:18.594184 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:18.595338 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:18.597450 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:18.598105 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:18.602849 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:18.623840 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 14:12:18.681349 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:18.707408 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:18.708997 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:18.710438 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:18.716179 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:18.718971 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:18.731472 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:18.732573 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:18.821517 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:18.825299 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:18.825629 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:18.827153 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:18.831029 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:18.836012 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:18.838654 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:18.849862 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 14:12:18.939361 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:18.939934 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:18.941479 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:18.941915 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:18.957127 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:12:18.965844 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 14:12:18.973716 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:18.974431 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:19.048934 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:19.052278 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:19.052574 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:19.061342 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:12:19.075734 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 14:12:19.078505 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:19.079468 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:19.088286 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:19.124272 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:19.127413 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:19.128994 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:19.144299 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:19.145066 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:19.154042 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:19.161113 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:19.170256 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:19.230669 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:19.231492 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:19.236241 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:19.238838 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:19.246274 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:19.260761 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:19.274477 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:19.312682 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:19.314491 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:19.322468 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:19.323240 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:19.327802 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:19.339695 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:19.339815 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:19.384907 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 14:12:19.427196 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:19.427994 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:19.431537 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:19.440323 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:19.441632 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:19.443472 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:19.451961 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:19.580848 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:19.582530 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:19.586655 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:19.595424 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:19.596239 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:19.598253 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:19.613832 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:19.681860 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 14:12:19.697620 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:19.704102 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:19.709306 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:19.709381 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:19.709546 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:19.710112 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:19.732967 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:19.764694 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 14:12:19.765615 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:19.800608 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:19.801442 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:19.805842 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:19.809303 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:19.811237 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:19.811837 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:19.834070 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 14:12:19.843220 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:19.905403 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 14:12:19.929955 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:19.933789 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:19.934728 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:19.936409 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:19.939087 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:19.951754 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:19.952692 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:20.011364 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 14:12:20.057547 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:20.062160 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:20.063746 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:20.063809 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:20.070911 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:20.083904 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:20.086534 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:20.101426 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 14:12:20.102306 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:20.103639 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:20.106912 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:20.116514 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:20.116935 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:20.126415 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:20.129863 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 14:12:20.135236 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:20.237533 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:20.244167 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:20.247725 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:20.254297 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:20.255152 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:20.255190 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:20.258957 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:20.275698 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:20.287549 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:20.293013 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:20.294528 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:20.296952 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:20.297057 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:20.315171 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 14:12:20.318027 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:20.337203 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:20.354866 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 14:12:20.408488 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:20.409386 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:20.416149 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:20.424592 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:20.427120 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:20.428076 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:20.435231 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:20.440320 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 14:12:20.627697 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:20.630836 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:20.631865 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:20.633682 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:20.635459 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:20.653039 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:20.654830 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:20.761027 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:20.764212 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:20.765372 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:20.769702 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:20.773028 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:20.795171 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:20.796038 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:20.798087 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:20.798283 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:20.806793 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:20.807330 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:20.811461 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:20.825263 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:20.831384 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:20.837790 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:20.934167 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:20.934237 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:20.938481 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:20.940124 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:20.943799 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:20.948565 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:20.977066 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:21.043370 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 14:12:21.056152 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:21.059190 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:21.065780 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:21.070727 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:21.080281 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:21.081764 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:21.087821 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:21.166430 19223 maintenance_manager.cc:419] P e7ca182af9f547e7b7200a15751811d3: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033394
W20240501 14:12:21.180567 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 14:12:21.181787 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:21.182832 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:21.199133 19222 maintenance_manager.cc:643] P e7ca182af9f547e7b7200a15751811d3: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.032s user 0.018s sys 0.005s Metrics: {"bytes_written":19949,"cfile_init":1,"dirs.queue_time_us":4169,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":664,"lbm_writes_lt_1ms":28,"peak_mem_usage":0,"rows_written":43,"thread_start_us":153,"threads_started":2,"wal-append.queue_time_us":346}
I20240501 14:12:21.203008 19223 maintenance_manager.cc:419] P e7ca182af9f547e7b7200a15751811d3: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 7323 bytes on disk
I20240501 14:12:21.203787 19222 maintenance_manager.cc:643] P e7ca182af9f547e7b7200a15751811d3: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4}
I20240501 14:12:21.209528 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:21.213842 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:21.215675 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:21.218281 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:21.231593 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:21.234297 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:21.250355 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:21.255223 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 14:12:21.256361 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:21.265518 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:21.266947 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:21.266943 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:21.275769 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:12:21.320676 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:21.364892 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:21.366098 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:21.370316 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:21.371308 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:21.384136 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:21.388772 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:21.393748 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:21.446681 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:21.449163 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:21.451889 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:21.451992 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:21.464118 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:21.465402 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:21.467094 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:12:21.590570 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:12:21.603811 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 14:12:21.604475 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 14:12:21.706876 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:21.708987 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:21.709656 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:21.710904 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:21.711130 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:21.740235 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:21.742861 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:21.759148 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:12:21.786370 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:21.839210 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:21.843907 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:21.846721 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:21.848474 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:21.856894 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:21.864213 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:21.868350 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:21.879528 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:21.879674 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:21.881297 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:21.884220 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:21.884606 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:12:21.888670 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:12:21.909090 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:21.926705 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:22.015914 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:22.020206 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:22.023802 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:22.025835 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:22.026129 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:22.035238 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:22.038632 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:22.128515 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 14:12:22.129763 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:22.132273 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:22.136359 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:22.140100 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:22.140818 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:22.156347 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:22.159108 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:22.249467 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 14:12:22.251463 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:22.255460 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:22.256196 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:22.256292 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:22.262594 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:22.264276 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:22.287902 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:22.309765 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:22.316529 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:22.318768 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:22.320775 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:12:22.321307 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:22.324018 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:22.329339 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:22.358271 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:22.437330 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:22.442381 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:22.443333 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:22.449331 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:22.454692 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 14:12:22.455418 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:22.456315 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:22.468436 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:22.520020 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:22.520859 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:22.522632 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:22.523262 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:22.529301 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:22.550578 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:22.562784 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:22.564335 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:12:22.601917 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:12:22.636592 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:12:22.652251 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 14:12:22.767012 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:22.771384 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:22.776405 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:22.778228 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:22.781327 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:22.785945 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:22.791600 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:22.815590 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:12:22.841512 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:12:22.851526 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:22.889624 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:22.889909 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:22.897509 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:22.901103 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:12:22.917479 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:22.918406 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:22.924299 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:22.930608 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:22.933723 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:22.934275 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:22.936103 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:22.938656 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:22.947204 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:22.948187 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:22.988463 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:23.071794 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:23.071977 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:23.077468 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:23.079196 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:23.082548 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:23.091213 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:23.091233 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:23.187135 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:23.189774 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:23.190290 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:23.192127 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:23.200520 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:23.203858 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:23.229612 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:23.286584 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:23.303906 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:23.310423 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:23.313359 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:23.314471 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:23.315186 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:23.317364 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:23.335728 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:23.392568 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:23.400034 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:23.400403 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:23.406571 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:23.411299 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:23.413909 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:23.436929 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:23.484061 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:23.493456 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:23.493836 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:23.495803 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:23.511101 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:23.519178 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:23.519822 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:23.570688 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:23.579975 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:23.580845 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:23.582280 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:23.584270 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:23.585737 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:23.589848 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:23.631717 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 14:12:23.632390 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:12:23.730022 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:23.819226 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:23.822366 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:23.824151 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:23.827378 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:23.827533 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:23.831957 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:23.833096 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:23.881700 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:12:23.915529 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 14:12:23.948091 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:23.954952 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:23.958788 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:23.959949 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:23.971344 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:23.977982 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:23.990428 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:23.994467 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:24.008159 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 14:12:24.012698 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:24.017661 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:24.029440 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:24.031558 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:24.031996 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:24.035157 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:24.071842 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 14:12:24.125919 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:24.131219 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:24.131587 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:24.131614 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:24.141419 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:24.143190 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:24.144474 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:24.233237 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:24.237344 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:24.238734 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:24.243736 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:24.247668 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:24.255964 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:24.261960 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:24.269587 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:24.358577 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:24.360215 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:24.362656 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 14:12:24.367799 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:24.368422 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:24.372449 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:24.375098 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:24.411428 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:24.433311 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:24.434468 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:24.436239 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:24.439319 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:24.446236 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:24.447764 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:24.448040 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:24.515374 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 14:12:24.521868 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:24.526890 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:24.530011 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:24.530658 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:24.531440 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:24.567502 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:24.569741 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:24.621711 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:24.624294 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:24.629354 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:24.632681 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:24.636359 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:24.637239 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:24.653178 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 14:12:24.669458 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:24.696202 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:24.861761 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:24.866834 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:24.871203 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:24.873080 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:24.879556 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:24.887030 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:24.892308 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:24.943915 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 14:12:24.985042 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:24.992985 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:24.996591 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:25.003190 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:25.016260 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:12:25.016995 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 14:12:25.020534 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:25.028911 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:25.033277 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:25.036168 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:25.061864 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:25.062959 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:25.068194 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:25.069629 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:25.073150 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:25.081547 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:25.082199 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:12:25.143785 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:12:25.152655 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 14:12:25.172638 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:25.177244 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:25.178442 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:25.180536 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:25.185596 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:25.195606 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:25.203662 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:25.210700 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:25.291227 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:25.292894 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:25.299224 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:25.303982 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:25.309263 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:25.311498 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:25.317057 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:25.387771 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:25.392352 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:25.402603 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:25.414129 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:25.416172 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:25.436026 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:25.439402 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:25.492396 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:25.493638 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:25.498775 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:25.499117 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:25.501574 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:25.529551 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:25.531332 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:12:25.564185 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 14:12:25.570375 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:25.586514 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:25.587898 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:25.590348 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:25.591810 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:25.592502 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:25.600109 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:25.654942 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:25.676784 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:25.679232 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:25.680666 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:25.683374 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:25.691184 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:25.695365 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:25.706457 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:25.889468 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 14:12:25.902478 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:25.904450 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:25.905730 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:25.911834 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:25.920774 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:25.931259 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:25.931636 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:26.040966 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:26.041496 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:26.043262 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:26.047276 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:26.059577 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:26.061475 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:26.063725 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:26.090623 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:26.091966 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:26.093173 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:26.093605 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:26.110404 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:26.110421 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:26.122613 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:26.214385 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:26.216243 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:26.224830 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:26.235018 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:26.236377 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:26.236909 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:26.238502 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:26.241715 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:26.263648 19554 maintenance_manager.cc:419] P 99a0be6ef2144b7fb247ba9d3f051d66: Scheduling FlushMRSOp(55830424a98348e382759c54afdeabcd): perf score=0.033334
I20240501 14:12:26.291270 19444 maintenance_manager.cc:643] P 99a0be6ef2144b7fb247ba9d3f051d66: FlushMRSOp(55830424a98348e382759c54afdeabcd) complete. Timing: real 0.027s user 0.026s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":184,"dirs.queue_time_us":155,"dirs.run_cpu_time_us":196,"dirs.run_wall_time_us":1093,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":480,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":306,"threads_started":3,"wal-append.queue_time_us":437}
I20240501 14:12:26.292418 19554 maintenance_manager.cc:419] P 99a0be6ef2144b7fb247ba9d3f051d66: Scheduling UndoDeltaBlockGCOp(55830424a98348e382759c54afdeabcd): 4581 bytes on disk
I20240501 14:12:26.292901 19444 maintenance_manager.cc:643] P 99a0be6ef2144b7fb247ba9d3f051d66: UndoDeltaBlockGCOp(55830424a98348e382759c54afdeabcd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
I20240501 14:12:26.349469 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:26.355080 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:26.355943 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:26.360821 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:26.361276 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:26.369160 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:26.371626 19816 maintenance_manager.cc:419] P da3c4e948ff747678f18456c134d7ed4: Scheduling FlushMRSOp(1d408b075d784953a57554bcdf19158a): perf score=0.033366
I20240501 14:12:26.392568 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:26.443712 19706 maintenance_manager.cc:643] P da3c4e948ff747678f18456c134d7ed4: FlushMRSOp(1d408b075d784953a57554bcdf19158a) complete. Timing: real 0.071s user 0.033s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":42172,"dirs.queue_time_us":12884,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":579,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":246,"threads_started":3,"wal-append.queue_time_us":156}
I20240501 14:12:26.445302 19816 maintenance_manager.cc:419] P da3c4e948ff747678f18456c134d7ed4: Scheduling UndoDeltaBlockGCOp(1d408b075d784953a57554bcdf19158a): 4921 bytes on disk
I20240501 14:12:26.446748 19706 maintenance_manager.cc:643] P da3c4e948ff747678f18456c134d7ed4: UndoDeltaBlockGCOp(1d408b075d784953a57554bcdf19158a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240501 14:12:26.457207 19685 maintenance_manager.cc:419] P ec8cde17a00645e7b38d5d29f7425367: Scheduling FlushMRSOp(85593473500b4d5eaf9f8b0bf2296777): perf score=0.033386
I20240501 14:12:26.530381 19575 maintenance_manager.cc:643] P ec8cde17a00645e7b38d5d29f7425367: FlushMRSOp(85593473500b4d5eaf9f8b0bf2296777) complete. Timing: real 0.073s user 0.034s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":186,"dirs.queue_time_us":22222,"dirs.run_cpu_time_us":223,"dirs.run_wall_time_us":991,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":696,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":280,"threads_started":3,"wal-append.queue_time_us":22514}
I20240501 14:12:26.531859 19685 maintenance_manager.cc:419] P ec8cde17a00645e7b38d5d29f7425367: Scheduling UndoDeltaBlockGCOp(85593473500b4d5eaf9f8b0bf2296777): 4834 bytes on disk
I20240501 14:12:26.532583 19575 maintenance_manager.cc:643] P ec8cde17a00645e7b38d5d29f7425367: UndoDeltaBlockGCOp(85593473500b4d5eaf9f8b0bf2296777) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
W20240501 14:12:26.543617 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:12:26.555305 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 14:12:26.556087 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:12:26.577376 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 14:12:26.601702 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:26.602061 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 14:12:26.604909 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:26.616786 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:26.617440 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:26.618033 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:26.618330 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:26.636795 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:26.639572 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:26.664647 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:26.670215 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:26.692960 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:26.693240 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:26.703745 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:26.714100 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:26.720896 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:26.784009 19685 maintenance_manager.cc:419] P ec8cde17a00645e7b38d5d29f7425367: Scheduling FlushMRSOp(661aaf83e3474966a257fcc039ef4784): perf score=0.033357
I20240501 14:12:26.795022 19554 maintenance_manager.cc:419] P 99a0be6ef2144b7fb247ba9d3f051d66: Scheduling FlushMRSOp(dab5e5722e2e4b0eaea73288d8dce265): perf score=0.033358
I20240501 14:12:26.813338 19575 maintenance_manager.cc:643] P ec8cde17a00645e7b38d5d29f7425367: FlushMRSOp(661aaf83e3474966a257fcc039ef4784) complete. Timing: real 0.029s user 0.019s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":90,"dirs.run_cpu_time_us":186,"dirs.run_wall_time_us":1719,"drs_written":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4,"lbm_write_time_us":551,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":106,"threads_started":1,"wal-append.queue_time_us":3116}
I20240501 14:12:26.814241 19685 maintenance_manager.cc:419] P ec8cde17a00645e7b38d5d29f7425367: Scheduling FlushMRSOp(dadab886b03540579d138cdb5b25e286): perf score=0.033364
I20240501 14:12:26.816834 19444 maintenance_manager.cc:643] P 99a0be6ef2144b7fb247ba9d3f051d66: FlushMRSOp(dab5e5722e2e4b0eaea73288d8dce265) complete. Timing: real 0.022s user 0.015s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":4142,"dirs.run_cpu_time_us":215,"dirs.run_wall_time_us":2064,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":575,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":162,"threads_started":2,"wal-append.queue_time_us":4620}
I20240501 14:12:26.817596 19554 maintenance_manager.cc:419] P 99a0be6ef2144b7fb247ba9d3f051d66: Scheduling UndoDeltaBlockGCOp(dab5e5722e2e4b0eaea73288d8dce265): 4921 bytes on disk
I20240501 14:12:26.818010 19444 maintenance_manager.cc:643] P 99a0be6ef2144b7fb247ba9d3f051d66: UndoDeltaBlockGCOp(dab5e5722e2e4b0eaea73288d8dce265) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240501 14:12:26.846006 19575 maintenance_manager.cc:643] P ec8cde17a00645e7b38d5d29f7425367: FlushMRSOp(dadab886b03540579d138cdb5b25e286) complete. Timing: real 0.031s user 0.019s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":43,"dirs.run_cpu_time_us":176,"dirs.run_wall_time_us":1326,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2747,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":112,"threads_started":1,"wal-append.queue_time_us":8078}
I20240501 14:12:26.846966 19685 maintenance_manager.cc:419] P ec8cde17a00645e7b38d5d29f7425367: Scheduling UndoDeltaBlockGCOp(dadab886b03540579d138cdb5b25e286): 4834 bytes on disk
I20240501 14:12:26.847484 19575 maintenance_manager.cc:643] P ec8cde17a00645e7b38d5d29f7425367: UndoDeltaBlockGCOp(dadab886b03540579d138cdb5b25e286) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20240501 14:12:26.848140 19685 maintenance_manager.cc:419] P ec8cde17a00645e7b38d5d29f7425367: Scheduling UndoDeltaBlockGCOp(661aaf83e3474966a257fcc039ef4784): 4581 bytes on disk
I20240501 14:12:26.848500 19575 maintenance_manager.cc:643] P ec8cde17a00645e7b38d5d29f7425367: UndoDeltaBlockGCOp(661aaf83e3474966a257fcc039ef4784) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240501 14:12:26.853641 19423 maintenance_manager.cc:419] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Scheduling FlushMRSOp(dadab886b03540579d138cdb5b25e286): perf score=0.033376
I20240501 14:12:26.892936 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:26.901655 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:26.901733 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:26.902321 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:26.922011 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:26.937608 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:12:26.940369 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:26.942667 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:26.942704 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:26.948029 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:26.948809 19816 maintenance_manager.cc:419] P da3c4e948ff747678f18456c134d7ed4: Scheduling FlushMRSOp(dab5e5722e2e4b0eaea73288d8dce265): perf score=0.033402
I20240501 14:12:26.965852 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:26.978427 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:26.987790 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:26.994717 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:27.008931 19706 maintenance_manager.cc:643] P da3c4e948ff747678f18456c134d7ed4: FlushMRSOp(dab5e5722e2e4b0eaea73288d8dce265) complete. Timing: real 0.060s user 0.013s sys 0.003s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":29600,"dirs.run_cpu_time_us":210,"dirs.run_wall_time_us":2747,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":512,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":186,"threads_started":2,"wal-append.queue_time_us":16645}
I20240501 14:12:27.009840 19816 maintenance_manager.cc:419] P da3c4e948ff747678f18456c134d7ed4: Scheduling UndoDeltaBlockGCOp(dab5e5722e2e4b0eaea73288d8dce265): 4921 bytes on disk
I20240501 14:12:27.010282 19706 maintenance_manager.cc:643] P da3c4e948ff747678f18456c134d7ed4: UndoDeltaBlockGCOp(dab5e5722e2e4b0eaea73288d8dce265) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
I20240501 14:12:27.010974 19816 maintenance_manager.cc:419] P da3c4e948ff747678f18456c134d7ed4: Scheduling FlushMRSOp(661aaf83e3474966a257fcc039ef4784): perf score=0.033418
I20240501 14:12:27.011623 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:27.025241 19313 maintenance_manager.cc:643] P fb1e6bc2d07b4ce3b622a66f14c93bdc: FlushMRSOp(dadab886b03540579d138cdb5b25e286) complete. Timing: real 0.171s user 0.034s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":16659,"dirs.queue_time_us":9737,"dirs.run_cpu_time_us":169,"dirs.run_wall_time_us":2459,"drs_written":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4,"lbm_write_time_us":543,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":1088,"threads_started":3,"wal-append.queue_time_us":310}
I20240501 14:12:27.038163 19423 maintenance_manager.cc:419] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Scheduling UndoDeltaBlockGCOp(dadab886b03540579d138cdb5b25e286): 4834 bytes on disk
I20240501 14:12:27.038118 19706 maintenance_manager.cc:643] P da3c4e948ff747678f18456c134d7ed4: FlushMRSOp(661aaf83e3474966a257fcc039ef4784) complete. Timing: real 0.027s user 0.016s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":2764,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":634,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":98,"threads_started":1,"wal-append.queue_time_us":196}
I20240501 14:12:27.038828 19313 maintenance_manager.cc:643] P fb1e6bc2d07b4ce3b622a66f14c93bdc: UndoDeltaBlockGCOp(dadab886b03540579d138cdb5b25e286) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
I20240501 14:12:27.038940 19816 maintenance_manager.cc:419] P da3c4e948ff747678f18456c134d7ed4: Scheduling UndoDeltaBlockGCOp(661aaf83e3474966a257fcc039ef4784): 4581 bytes on disk
I20240501 14:12:27.039479 19706 maintenance_manager.cc:643] P da3c4e948ff747678f18456c134d7ed4: UndoDeltaBlockGCOp(661aaf83e3474966a257fcc039ef4784) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
W20240501 14:12:27.047004 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:12:27.062566 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:12:27.089560 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:12:27.213629 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:12:27.281255 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:12:27.298825 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 14:12:27.319633 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:27.329782 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:27.331322 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:27.335333 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:27.347504 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:27.348356 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:27.350534 19685 maintenance_manager.cc:419] P ec8cde17a00645e7b38d5d29f7425367: Scheduling FlushMRSOp(db0f01ee72b24ec9b84410cf94ef9759): perf score=0.033338
I20240501 14:12:27.359696 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:27.362385 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:27.363538 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:27.373538 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:12:27.379653 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 14:12:27.394438 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:27.407516 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:27.409590 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:27.415197 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:27.425379 19575 maintenance_manager.cc:643] P ec8cde17a00645e7b38d5d29f7425367: FlushMRSOp(db0f01ee72b24ec9b84410cf94ef9759) complete. Timing: real 0.075s user 0.016s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":26293,"dirs.run_cpu_time_us":232,"dirs.run_wall_time_us":10622,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":522,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":145,"threads_started":2,"wal-append.queue_time_us":26616}
I20240501 14:12:27.426234 19685 maintenance_manager.cc:419] P ec8cde17a00645e7b38d5d29f7425367: Scheduling UndoDeltaBlockGCOp(db0f01ee72b24ec9b84410cf94ef9759): 4581 bytes on disk
I20240501 14:12:27.426632 19575 maintenance_manager.cc:643] P ec8cde17a00645e7b38d5d29f7425367: UndoDeltaBlockGCOp(db0f01ee72b24ec9b84410cf94ef9759) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20240501 14:12:27.514362 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:27.525166 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:27.530210 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:27.531508 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:27.531884 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:27.535250 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:27.541165 19423 maintenance_manager.cc:419] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Scheduling FlushMRSOp(78e14dea8b6e4c0abd1c88dcd5b4e608): perf score=0.033392
I20240501 14:12:27.545655 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:27.564649 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:27.570482 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:27.571518 19554 maintenance_manager.cc:419] P 99a0be6ef2144b7fb247ba9d3f051d66: Scheduling FlushMRSOp(78e14dea8b6e4c0abd1c88dcd5b4e608): perf score=0.033397
I20240501 14:12:27.588284 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:27.591559 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:27.604827 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:27.605854 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:27.619107 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:27.635027 19444 maintenance_manager.cc:643] P 99a0be6ef2144b7fb247ba9d3f051d66: FlushMRSOp(78e14dea8b6e4c0abd1c88dcd5b4e608) complete. Timing: real 0.063s user 0.017s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":19405,"dirs.run_cpu_time_us":197,"dirs.run_wall_time_us":1281,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1116,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":328,"threads_started":2,"wal-append.queue_time_us":9266}
I20240501 14:12:27.636011 19554 maintenance_manager.cc:419] P 99a0be6ef2144b7fb247ba9d3f051d66: Scheduling UndoDeltaBlockGCOp(78e14dea8b6e4c0abd1c88dcd5b4e608): 4921 bytes on disk
I20240501 14:12:27.637894 19444 maintenance_manager.cc:643] P 99a0be6ef2144b7fb247ba9d3f051d66: UndoDeltaBlockGCOp(78e14dea8b6e4c0abd1c88dcd5b4e608) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20240501 14:12:27.637892 19313 maintenance_manager.cc:643] P fb1e6bc2d07b4ce3b622a66f14c93bdc: FlushMRSOp(78e14dea8b6e4c0abd1c88dcd5b4e608) complete. Timing: real 0.097s user 0.017s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":65008,"dirs.run_cpu_time_us":243,"dirs.run_wall_time_us":4971,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":905,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":150,"threads_started":2,"wal-append.queue_time_us":11956}
I20240501 14:12:27.638742 19423 maintenance_manager.cc:419] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Scheduling UndoDeltaBlockGCOp(78e14dea8b6e4c0abd1c88dcd5b4e608): 4921 bytes on disk
I20240501 14:12:27.639901 19313 maintenance_manager.cc:643] P fb1e6bc2d07b4ce3b622a66f14c93bdc: UndoDeltaBlockGCOp(78e14dea8b6e4c0abd1c88dcd5b4e608) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20240501 14:12:27.640591 19423 maintenance_manager.cc:419] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Scheduling FlushMRSOp(410a7dd0dd734684a82875912966657d): perf score=0.033419
I20240501 14:12:27.660734 19313 maintenance_manager.cc:643] P fb1e6bc2d07b4ce3b622a66f14c93bdc: FlushMRSOp(410a7dd0dd734684a82875912966657d) complete. Timing: real 0.020s user 0.010s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":38,"dirs.run_cpu_time_us":160,"dirs.run_wall_time_us":1194,"drs_written":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4,"lbm_write_time_us":475,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":98,"threads_started":1,"wal-append.queue_time_us":228}
I20240501 14:12:27.661516 19423 maintenance_manager.cc:419] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Scheduling UndoDeltaBlockGCOp(410a7dd0dd734684a82875912966657d): 4832 bytes on disk
I20240501 14:12:27.661897 19313 maintenance_manager.cc:643] P fb1e6bc2d07b4ce3b622a66f14c93bdc: UndoDeltaBlockGCOp(410a7dd0dd734684a82875912966657d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
W20240501 14:12:27.708158 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 14:12:27.716909 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:27.721091 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:27.721742 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:27.722023 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:27.724942 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:27.727737 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:27.745476 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:27.770731 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:12:27.780437 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 14:12:27.845525 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:27.846386 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:27.848929 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:27.852510 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:27.856146 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:27.858594 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:27.880398 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:27.891551 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 14:12:27.913087 19423 maintenance_manager.cc:419] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Scheduling FlushMRSOp(ffb4b6a2c19b497f8117fbbc9ea50082): perf score=0.033336
I20240501 14:12:27.928018 19313 maintenance_manager.cc:643] P fb1e6bc2d07b4ce3b622a66f14c93bdc: FlushMRSOp(ffb4b6a2c19b497f8117fbbc9ea50082) complete. Timing: real 0.015s user 0.013s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":56,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":1393,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":508,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":96,"threads_started":1,"wal-append.queue_time_us":165}
I20240501 14:12:27.928778 19685 maintenance_manager.cc:419] P ec8cde17a00645e7b38d5d29f7425367: Scheduling FlushMRSOp(ffb4b6a2c19b497f8117fbbc9ea50082): perf score=0.033342
I20240501 14:12:27.928882 19423 maintenance_manager.cc:419] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Scheduling UndoDeltaBlockGCOp(ffb4b6a2c19b497f8117fbbc9ea50082): 4921 bytes on disk
I20240501 14:12:27.929286 19313 maintenance_manager.cc:643] P fb1e6bc2d07b4ce3b622a66f14c93bdc: UndoDeltaBlockGCOp(ffb4b6a2c19b497f8117fbbc9ea50082) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240501 14:12:27.929893 19423 maintenance_manager.cc:419] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Scheduling FlushMRSOp(ec2a46f8cfb3406082970e5fdb59b101): perf score=0.033339
I20240501 14:12:27.948199 19313 maintenance_manager.cc:643] P fb1e6bc2d07b4ce3b622a66f14c93bdc: FlushMRSOp(ec2a46f8cfb3406082970e5fdb59b101) complete. Timing: real 0.018s user 0.016s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":39,"dirs.run_cpu_time_us":148,"dirs.run_wall_time_us":1248,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":571,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":90,"threads_started":1,"wal-append.queue_time_us":890}
I20240501 14:12:27.949195 19423 maintenance_manager.cc:419] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Scheduling UndoDeltaBlockGCOp(ec2a46f8cfb3406082970e5fdb59b101): 4832 bytes on disk
I20240501 14:12:27.949620 19313 maintenance_manager.cc:643] P fb1e6bc2d07b4ce3b622a66f14c93bdc: UndoDeltaBlockGCOp(ec2a46f8cfb3406082970e5fdb59b101) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4}
I20240501 14:12:27.951294 19575 maintenance_manager.cc:643] P ec8cde17a00645e7b38d5d29f7425367: FlushMRSOp(ffb4b6a2c19b497f8117fbbc9ea50082) complete. Timing: real 0.022s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":208,"dirs.run_cpu_time_us":160,"dirs.run_wall_time_us":3607,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":594,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":135,"threads_started":2,"wal-append.queue_time_us":142}
I20240501 14:12:27.952145 19685 maintenance_manager.cc:419] P ec8cde17a00645e7b38d5d29f7425367: Scheduling UndoDeltaBlockGCOp(ffb4b6a2c19b497f8117fbbc9ea50082): 4921 bytes on disk
I20240501 14:12:27.952530 19575 maintenance_manager.cc:643] P ec8cde17a00645e7b38d5d29f7425367: UndoDeltaBlockGCOp(ffb4b6a2c19b497f8117fbbc9ea50082) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
I20240501 14:12:27.983083 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:27.990862 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:27.992928 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:28.004345 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:28.010919 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:28.011946 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:28.012946 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:12:28.019279 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:28.023638 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:28.040766 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 14:12:28.043035 19816 maintenance_manager.cc:419] P da3c4e948ff747678f18456c134d7ed4: Scheduling FlushMRSOp(b6be9a115fe14c098c29cad819849f95): perf score=0.033373
I20240501 14:12:28.047375 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:28.049736 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:28.052227 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:28.072538 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:28.073830 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:28.078058 19706 maintenance_manager.cc:643] P da3c4e948ff747678f18456c134d7ed4: FlushMRSOp(b6be9a115fe14c098c29cad819849f95) complete. Timing: real 0.035s user 0.015s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"drs_written":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4,"lbm_write_time_us":575,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":5650,"threads_started":2,"wal-append.queue_time_us":10237}
I20240501 14:12:28.078922 19816 maintenance_manager.cc:419] P da3c4e948ff747678f18456c134d7ed4: Scheduling UndoDeltaBlockGCOp(b6be9a115fe14c098c29cad819849f95): 4581 bytes on disk
I20240501 14:12:28.080884 19706 maintenance_manager.cc:643] P da3c4e948ff747678f18456c134d7ed4: UndoDeltaBlockGCOp(b6be9a115fe14c098c29cad819849f95) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
I20240501 14:12:28.081729 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:28.140170 19554 maintenance_manager.cc:419] P 99a0be6ef2144b7fb247ba9d3f051d66: Scheduling FlushMRSOp(b6be9a115fe14c098c29cad819849f95): perf score=0.033396
I20240501 14:12:28.154934 19444 maintenance_manager.cc:643] P 99a0be6ef2144b7fb247ba9d3f051d66: FlushMRSOp(b6be9a115fe14c098c29cad819849f95) complete. Timing: real 0.014s user 0.012s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":148,"dirs.run_cpu_time_us":167,"dirs.run_wall_time_us":1492,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":697,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":909,"threads_started":2,"wal-append.queue_time_us":905}
I20240501 14:12:28.155732 19554 maintenance_manager.cc:419] P 99a0be6ef2144b7fb247ba9d3f051d66: Scheduling UndoDeltaBlockGCOp(b6be9a115fe14c098c29cad819849f95): 4581 bytes on disk
I20240501 14:12:28.156082 19444 maintenance_manager.cc:643] P 99a0be6ef2144b7fb247ba9d3f051d66: UndoDeltaBlockGCOp(b6be9a115fe14c098c29cad819849f95) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4}
W20240501 14:12:28.156307 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 14:12:28.173986 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:28.176895 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:28.177349 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:28.183264 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:28.187772 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:28.191752 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:28.215683 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:28.328547 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:28.363041 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:28.371209 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:28.372135 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:28.378808 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:28.379957 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:28.398494 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:28.398784 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:28.454421 19685 maintenance_manager.cc:419] P ec8cde17a00645e7b38d5d29f7425367: Scheduling FlushMRSOp(cf62635e9a994259b9e293e5eb425b55): perf score=0.033367
I20240501 14:12:28.472970 19575 maintenance_manager.cc:643] P ec8cde17a00645e7b38d5d29f7425367: FlushMRSOp(cf62635e9a994259b9e293e5eb425b55) complete. Timing: real 0.018s user 0.011s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":273,"dirs.run_cpu_time_us":206,"dirs.run_wall_time_us":1526,"drs_written":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4,"lbm_write_time_us":530,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":152,"threads_started":2,"wal-append.queue_time_us":126}
I20240501 14:12:28.473893 19685 maintenance_manager.cc:419] P ec8cde17a00645e7b38d5d29f7425367: Scheduling UndoDeltaBlockGCOp(cf62635e9a994259b9e293e5eb425b55): 4834 bytes on disk
I20240501 14:12:28.474285 19575 maintenance_manager.cc:643] P ec8cde17a00645e7b38d5d29f7425367: UndoDeltaBlockGCOp(cf62635e9a994259b9e293e5eb425b55) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20240501 14:12:28.511531 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:28.512845 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:28.516237 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:28.521270 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:28.527524 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:28.538045 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:28.550364 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:28.550587 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:28.560703 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:28.571336 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:28.591037 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:28.591638 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:28.591941 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:28.608950 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:12:28.667196 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240501 14:12:28.695746 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:28.704747 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:28.710618 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:28.720922 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:28.724237 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:28.734565 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:28.736531 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:28.742079 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:28.751094 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:28.756760 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:28.758888 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:28.771215 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:28.793224 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:28.797530 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:28.898247 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:28.900838 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:28.901677 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:28.902253 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:28.908403 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:28.909559 19554 maintenance_manager.cc:419] P 99a0be6ef2144b7fb247ba9d3f051d66: Scheduling FlushMRSOp(98af8b06b9324e3fa7cdb898b7b9fab9): perf score=0.033355
I20240501 14:12:28.920070 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:28.931145 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:28.933363 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 14:12:28.938563 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:28.940919 19444 maintenance_manager.cc:643] P 99a0be6ef2144b7fb247ba9d3f051d66: FlushMRSOp(98af8b06b9324e3fa7cdb898b7b9fab9) complete. Timing: real 0.031s user 0.015s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":1681,"dirs.run_cpu_time_us":178,"dirs.run_wall_time_us":1650,"drs_written":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1740,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":184,"threads_started":2,"wal-append.queue_time_us":162}
I20240501 14:12:28.941728 19554 maintenance_manager.cc:419] P 99a0be6ef2144b7fb247ba9d3f051d66: Scheduling UndoDeltaBlockGCOp(98af8b06b9324e3fa7cdb898b7b9fab9): 4834 bytes on disk
I20240501 14:12:28.942176 19444 maintenance_manager.cc:643] P 99a0be6ef2144b7fb247ba9d3f051d66: UndoDeltaBlockGCOp(98af8b06b9324e3fa7cdb898b7b9fab9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
I20240501 14:12:28.954180 19423 maintenance_manager.cc:419] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Scheduling FlushMRSOp(12ffa74f7a5b472aac57253693273cad): perf score=0.033365
I20240501 14:12:28.968425 19313 maintenance_manager.cc:643] P fb1e6bc2d07b4ce3b622a66f14c93bdc: FlushMRSOp(12ffa74f7a5b472aac57253693273cad) complete. Timing: real 0.014s user 0.012s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":164,"dirs.run_cpu_time_us":236,"dirs.run_wall_time_us":1402,"drs_written":1,"lbm_read_time_us":40,"lbm_reads_lt_1ms":4,"lbm_write_time_us":526,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":143,"threads_started":2,"wal-append.queue_time_us":140}
I20240501 14:12:28.969262 19423 maintenance_manager.cc:419] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Scheduling UndoDeltaBlockGCOp(12ffa74f7a5b472aac57253693273cad): 4921 bytes on disk
I20240501 14:12:28.969605 19313 maintenance_manager.cc:643] P fb1e6bc2d07b4ce3b622a66f14c93bdc: UndoDeltaBlockGCOp(12ffa74f7a5b472aac57253693273cad) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4}
W20240501 14:12:28.995559 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 14:12:29.019708 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:29.022620 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:29.026044 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:29.037434 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:29.045002 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:29.045869 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:29.047468 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:12:29.053025 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 14:12:29.084371 19816 maintenance_manager.cc:419] P da3c4e948ff747678f18456c134d7ed4: Scheduling FlushMRSOp(12ffa74f7a5b472aac57253693273cad): perf score=0.033402
I20240501 14:12:29.099325 19706 maintenance_manager.cc:643] P da3c4e948ff747678f18456c134d7ed4: FlushMRSOp(12ffa74f7a5b472aac57253693273cad) complete. Timing: real 0.015s user 0.014s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":182,"dirs.run_cpu_time_us":212,"dirs.run_wall_time_us":1645,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":488,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":148,"threads_started":2,"wal-append.queue_time_us":155}
I20240501 14:12:29.100133 19816 maintenance_manager.cc:419] P da3c4e948ff747678f18456c134d7ed4: Scheduling UndoDeltaBlockGCOp(12ffa74f7a5b472aac57253693273cad): 4921 bytes on disk
I20240501 14:12:29.100489 19706 maintenance_manager.cc:643] P da3c4e948ff747678f18456c134d7ed4: UndoDeltaBlockGCOp(12ffa74f7a5b472aac57253693273cad) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
I20240501 14:12:29.101125 19816 maintenance_manager.cc:419] P da3c4e948ff747678f18456c134d7ed4: Scheduling FlushMRSOp(bb291a0c743c4e638f79da36cc77faa3): perf score=0.033405
I20240501 14:12:29.115602 19706 maintenance_manager.cc:643] P da3c4e948ff747678f18456c134d7ed4: FlushMRSOp(bb291a0c743c4e638f79da36cc77faa3) complete. Timing: real 0.014s user 0.012s sys 0.001s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":53,"dirs.run_cpu_time_us":164,"dirs.run_wall_time_us":1410,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":443,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":76,"threads_started":1,"wal-append.queue_time_us":158}
I20240501 14:12:29.117158 19816 maintenance_manager.cc:419] P da3c4e948ff747678f18456c134d7ed4: Scheduling UndoDeltaBlockGCOp(bb291a0c743c4e638f79da36cc77faa3): 4581 bytes on disk
I20240501 14:12:29.117899 19706 maintenance_manager.cc:643] P da3c4e948ff747678f18456c134d7ed4: UndoDeltaBlockGCOp(bb291a0c743c4e638f79da36cc77faa3) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
I20240501 14:12:29.132067 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:29.132301 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:29.138159 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:29.143679 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:29.146775 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:29.151829 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 14:12:29.155722 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:29.173533 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:29.249994 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:29.252548 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:29.254129 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:29.254842 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:29.258471 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:29.272511 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:29.275228 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:29.359133 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:29.359697 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:29.361697 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:29.362469 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:29.374888 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:29.381621 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:29.391561 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:29.443989 19554 maintenance_manager.cc:419] P 99a0be6ef2144b7fb247ba9d3f051d66: Scheduling FlushMRSOp(a9edf1764b55420ea5d519e07ae1d5e0): perf score=0.033353
W20240501 14:12:29.453617 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:29.459225 19444 maintenance_manager.cc:643] P 99a0be6ef2144b7fb247ba9d3f051d66: FlushMRSOp(a9edf1764b55420ea5d519e07ae1d5e0) complete. Timing: real 0.015s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":155,"dirs.run_cpu_time_us":228,"dirs.run_wall_time_us":1603,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":486,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":145,"threads_started":2,"wal-append.queue_time_us":148}
I20240501 14:12:29.460273 19554 maintenance_manager.cc:419] P 99a0be6ef2144b7fb247ba9d3f051d66: Scheduling UndoDeltaBlockGCOp(a9edf1764b55420ea5d519e07ae1d5e0): 4581 bytes on disk
I20240501 14:12:29.461061 19444 maintenance_manager.cc:643] P 99a0be6ef2144b7fb247ba9d3f051d66: UndoDeltaBlockGCOp(a9edf1764b55420ea5d519e07ae1d5e0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4}
I20240501 14:12:29.471472 19423 maintenance_manager.cc:419] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Scheduling FlushMRSOp(d63afd081986473aa2715d074b123a93): perf score=0.033359
I20240501 14:12:29.477766 19685 maintenance_manager.cc:419] P ec8cde17a00645e7b38d5d29f7425367: Scheduling FlushMRSOp(a9edf1764b55420ea5d519e07ae1d5e0): perf score=0.033362
I20240501 14:12:29.486935 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:29.488744 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:29.491353 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:29.505842 19575 maintenance_manager.cc:643] P ec8cde17a00645e7b38d5d29f7425367: FlushMRSOp(a9edf1764b55420ea5d519e07ae1d5e0) complete. Timing: real 0.028s user 0.011s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":190,"dirs.run_cpu_time_us":208,"dirs.run_wall_time_us":1582,"drs_written":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1088,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":156,"threads_started":2,"wal-append.queue_time_us":149}
I20240501 14:12:29.506637 19685 maintenance_manager.cc:419] P ec8cde17a00645e7b38d5d29f7425367: Scheduling UndoDeltaBlockGCOp(a9edf1764b55420ea5d519e07ae1d5e0): 4581 bytes on disk
I20240501 14:12:29.507078 19575 maintenance_manager.cc:643] P ec8cde17a00645e7b38d5d29f7425367: UndoDeltaBlockGCOp(a9edf1764b55420ea5d519e07ae1d5e0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240501 14:12:29.512254 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:29.512511 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:29.521502 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:29.525571 19313 maintenance_manager.cc:643] P fb1e6bc2d07b4ce3b622a66f14c93bdc: FlushMRSOp(d63afd081986473aa2715d074b123a93) complete. Timing: real 0.054s user 0.014s sys 0.002s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":8954,"dirs.run_cpu_time_us":196,"dirs.run_wall_time_us":1256,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":508,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":975,"threads_started":2,"wal-append.queue_time_us":17823}
I20240501 14:12:29.527818 19423 maintenance_manager.cc:419] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Scheduling FlushMRSOp(ad7c255baee74e629e2976a942a4174f): perf score=0.033374
I20240501 14:12:29.535478 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:29.539858 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 14:12:29.556771 19313 maintenance_manager.cc:643] P fb1e6bc2d07b4ce3b622a66f14c93bdc: FlushMRSOp(ad7c255baee74e629e2976a942a4174f) complete. Timing: real 0.028s user 0.016s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":232,"dirs.run_cpu_time_us":221,"dirs.run_wall_time_us":1364,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":736,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":93,"threads_started":1,"wal-append.queue_time_us":4103}
I20240501 14:12:29.557528 19423 maintenance_manager.cc:419] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Scheduling UndoDeltaBlockGCOp(d63afd081986473aa2715d074b123a93): 4921 bytes on disk
I20240501 14:12:29.557955 19313 maintenance_manager.cc:643] P fb1e6bc2d07b4ce3b622a66f14c93bdc: UndoDeltaBlockGCOp(d63afd081986473aa2715d074b123a93) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4}
I20240501 14:12:29.559661 19423 maintenance_manager.cc:419] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Scheduling UndoDeltaBlockGCOp(ad7c255baee74e629e2976a942a4174f): 4834 bytes on disk
I20240501 14:12:29.560051 19313 maintenance_manager.cc:643] P fb1e6bc2d07b4ce3b622a66f14c93bdc: UndoDeltaBlockGCOp(ad7c255baee74e629e2976a942a4174f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4}
W20240501 14:12:29.562104 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 14:12:29.632733 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:29.633067 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:29.634493 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:29.641434 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:29.642290 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:29.648809 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:29.648983 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 14:12:29.663697 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:29.669279 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:29.673585 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:29.674651 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:29.675099 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:29.675350 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:29.686475 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:29.693604 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:29.847795 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 14:12:29.854676 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:29.856462 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:29.866022 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:29.867259 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:29.867727 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:29.868387 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:29.868629 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:29.878219 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:29.883975 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:29.889822 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:29.896354 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:29.904549 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:29.912418 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:12:29.920145 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:29.920233 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:29.925920 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:30.009408 19685 maintenance_manager.cc:419] P ec8cde17a00645e7b38d5d29f7425367: Scheduling FlushMRSOp(cc8e888d46d34a22b6d4409ec2ab53a9): perf score=0.033335
I20240501 14:12:30.024873 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:30.035928 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:30.035935 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:30.036643 19575 maintenance_manager.cc:643] P ec8cde17a00645e7b38d5d29f7425367: FlushMRSOp(cc8e888d46d34a22b6d4409ec2ab53a9) complete. Timing: real 0.027s user 0.019s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":5964,"dirs.run_cpu_time_us":245,"dirs.run_wall_time_us":2425,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":707,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":161,"threads_started":2,"wal-append.queue_time_us":2620}
I20240501 14:12:30.037755 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:30.038589 19685 maintenance_manager.cc:419] P ec8cde17a00645e7b38d5d29f7425367: Scheduling UndoDeltaBlockGCOp(cc8e888d46d34a22b6d4409ec2ab53a9): 4921 bytes on disk
I20240501 14:12:30.039300 19575 maintenance_manager.cc:643] P ec8cde17a00645e7b38d5d29f7425367: UndoDeltaBlockGCOp(cc8e888d46d34a22b6d4409ec2ab53a9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
I20240501 14:12:30.046113 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:30.062374 19423 maintenance_manager.cc:419] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Scheduling FlushMRSOp(0988c468757d49738765bdcb31794bd1): perf score=0.033344
I20240501 14:12:30.066468 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:30.068873 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:30.079464 19313 maintenance_manager.cc:643] P fb1e6bc2d07b4ce3b622a66f14c93bdc: FlushMRSOp(0988c468757d49738765bdcb31794bd1) complete. Timing: real 0.017s user 0.015s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":574,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":503,"threads_started":2,"wal-append.queue_time_us":145}
I20240501 14:12:30.080238 19423 maintenance_manager.cc:419] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Scheduling UndoDeltaBlockGCOp(0988c468757d49738765bdcb31794bd1): 4581 bytes on disk
I20240501 14:12:30.080637 19313 maintenance_manager.cc:643] P fb1e6bc2d07b4ce3b622a66f14c93bdc: UndoDeltaBlockGCOp(0988c468757d49738765bdcb31794bd1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240501 14:12:30.122135 19816 maintenance_manager.cc:419] P da3c4e948ff747678f18456c134d7ed4: Scheduling FlushMRSOp(0988c468757d49738765bdcb31794bd1): perf score=0.033364
I20240501 14:12:30.133945 19706 maintenance_manager.cc:643] P da3c4e948ff747678f18456c134d7ed4: FlushMRSOp(0988c468757d49738765bdcb31794bd1) complete. Timing: real 0.012s user 0.009s sys 0.001s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":171,"dirs.run_cpu_time_us":228,"dirs.run_wall_time_us":1493,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":472,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":141,"threads_started":2,"wal-append.queue_time_us":152}
I20240501 14:12:30.134785 19816 maintenance_manager.cc:419] P da3c4e948ff747678f18456c134d7ed4: Scheduling UndoDeltaBlockGCOp(0988c468757d49738765bdcb31794bd1): 4581 bytes on disk
I20240501 14:12:30.135174 19706 maintenance_manager.cc:643] P da3c4e948ff747678f18456c134d7ed4: UndoDeltaBlockGCOp(0988c468757d49738765bdcb31794bd1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
I20240501 14:12:30.214908 19554 maintenance_manager.cc:419] P 99a0be6ef2144b7fb247ba9d3f051d66: Scheduling FlushMRSOp(272eee3450cf4f48a4e1de2dcb28c0f7): perf score=0.033387
I20240501 14:12:30.232453 19444 maintenance_manager.cc:643] P 99a0be6ef2144b7fb247ba9d3f051d66: FlushMRSOp(272eee3450cf4f48a4e1de2dcb28c0f7) complete. Timing: real 0.017s user 0.016s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":2390,"drs_written":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4,"lbm_write_time_us":733,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":163,"threads_started":2,"wal-append.queue_time_us":128}
I20240501 14:12:30.234119 19554 maintenance_manager.cc:419] P 99a0be6ef2144b7fb247ba9d3f051d66: Scheduling UndoDeltaBlockGCOp(272eee3450cf4f48a4e1de2dcb28c0f7): 4834 bytes on disk
I20240501 14:12:30.234565 19444 maintenance_manager.cc:643] P 99a0be6ef2144b7fb247ba9d3f051d66: UndoDeltaBlockGCOp(272eee3450cf4f48a4e1de2dcb28c0f7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240501 14:12:30.238437 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:30.241854 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:30.242295 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:30.247186 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:30.257330 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:30.266733 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:30.268321 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:30.315814 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:12:30.317489 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 14:12:30.367012 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:30.368166 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:30.369807 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:30.379529 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 14:12:30.379871 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:30.412149 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:30.412818 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:30.416852 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:30.432704 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:30.433053 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:30.433444 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:30.439989 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:30.440025 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:30.457195 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:30.471093 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:30.474058 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:30.479650 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 14:12:30.560899 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:30.562177 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:30.563127 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:30.563252 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 14:12:30.570894 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:30.572510 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:30.573742 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:30.601346 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:30.639638 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 14:12:30.642926 19816 maintenance_manager.cc:419] P da3c4e948ff747678f18456c134d7ed4: Scheduling FlushMRSOp(11f64b59f83e458f8cf51fd5bcd36fee): perf score=0.033350
I20240501 14:12:30.659595 19706 maintenance_manager.cc:643] P da3c4e948ff747678f18456c134d7ed4: FlushMRSOp(11f64b59f83e458f8cf51fd5bcd36fee) complete. Timing: real 0.016s user 0.013s sys 0.001s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":242,"dirs.run_cpu_time_us":226,"dirs.run_wall_time_us":1600,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":679,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":123,"threads_started":2,"wal-append.queue_time_us":138}
I20240501 14:12:30.660387 19816 maintenance_manager.cc:419] P da3c4e948ff747678f18456c134d7ed4: Scheduling UndoDeltaBlockGCOp(11f64b59f83e458f8cf51fd5bcd36fee): 4834 bytes on disk
I20240501 14:12:30.660815 19706 maintenance_manager.cc:643] P da3c4e948ff747678f18456c134d7ed4: UndoDeltaBlockGCOp(11f64b59f83e458f8cf51fd5bcd36fee) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
I20240501 14:12:30.693259 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:30.693244 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:30.693533 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:30.694738 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:30.702008 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:30.709744 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:30.733935 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:30.736721 19554 maintenance_manager.cc:419] P 99a0be6ef2144b7fb247ba9d3f051d66: Scheduling FlushMRSOp(f6ca088ce3aa4b97aa54c8ce6d700921): perf score=0.033375
I20240501 14:12:30.751574 19444 maintenance_manager.cc:643] P 99a0be6ef2144b7fb247ba9d3f051d66: FlushMRSOp(f6ca088ce3aa4b97aa54c8ce6d700921) complete. Timing: real 0.012s user 0.011s sys 0.001s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":190,"dirs.run_cpu_time_us":163,"dirs.run_wall_time_us":1654,"drs_written":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4,"lbm_write_time_us":427,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":147,"threads_started":2,"wal-append.queue_time_us":172}
I20240501 14:12:30.752555 19554 maintenance_manager.cc:419] P 99a0be6ef2144b7fb247ba9d3f051d66: Scheduling UndoDeltaBlockGCOp(f6ca088ce3aa4b97aa54c8ce6d700921): 4581 bytes on disk
I20240501 14:12:30.754057 19444 maintenance_manager.cc:643] P 99a0be6ef2144b7fb247ba9d3f051d66: UndoDeltaBlockGCOp(f6ca088ce3aa4b97aa54c8ce6d700921) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240501 14:12:30.792021 19685 maintenance_manager.cc:419] P ec8cde17a00645e7b38d5d29f7425367: Scheduling FlushMRSOp(2dcca6e0e6374520bd4522c80c110e97): perf score=0.033389
I20240501 14:12:30.809897 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:30.815635 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:30.815723 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:30.819504 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:30.820856 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:30.823823 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:30.826813 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:30.871886 19575 maintenance_manager.cc:643] P ec8cde17a00645e7b38d5d29f7425367: FlushMRSOp(2dcca6e0e6374520bd4522c80c110e97) complete. Timing: real 0.080s user 0.020s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":256,"dirs.run_cpu_time_us":192,"dirs.run_wall_time_us":3636,"drs_written":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4,"lbm_write_time_us":705,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":195,"threads_started":2,"wal-append.queue_time_us":182}
I20240501 14:12:30.873085 19685 maintenance_manager.cc:419] P ec8cde17a00645e7b38d5d29f7425367: Scheduling UndoDeltaBlockGCOp(2dcca6e0e6374520bd4522c80c110e97): 4920 bytes on disk
I20240501 14:12:30.874065 19575 maintenance_manager.cc:643] P ec8cde17a00645e7b38d5d29f7425367: UndoDeltaBlockGCOp(2dcca6e0e6374520bd4522c80c110e97) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20240501 14:12:30.917593 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:30.921758 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:30.924852 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:30.926700 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:30.932977 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:30.933784 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:30.940641 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:30.998831 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:31.002857 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:31.005205 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:31.007491 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:31.008580 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:31.012842 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:31.017490 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:31.034284 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:31.087543 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:31.095274 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:31.095432 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:31.097151 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:31.099004 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:31.102970 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:31.120607 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:31.256070 19554 maintenance_manager.cc:419] P 99a0be6ef2144b7fb247ba9d3f051d66: Scheduling FlushMRSOp(7eea58d4a33f422a9ada1e82d408577f): perf score=0.033341
I20240501 14:12:31.278261 19444 maintenance_manager.cc:643] P 99a0be6ef2144b7fb247ba9d3f051d66: FlushMRSOp(7eea58d4a33f422a9ada1e82d408577f) complete. Timing: real 0.022s user 0.017s sys 0.001s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":1721,"dirs.run_cpu_time_us":237,"dirs.run_wall_time_us":1401,"drs_written":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4,"lbm_write_time_us":616,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":133,"threads_started":2,"wal-append.queue_time_us":136}
I20240501 14:12:31.279279 19554 maintenance_manager.cc:419] P 99a0be6ef2144b7fb247ba9d3f051d66: Scheduling UndoDeltaBlockGCOp(7eea58d4a33f422a9ada1e82d408577f): 4581 bytes on disk
I20240501 14:12:31.279738 19444 maintenance_manager.cc:643] P 99a0be6ef2144b7fb247ba9d3f051d66: UndoDeltaBlockGCOp(7eea58d4a33f422a9ada1e82d408577f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
I20240501 14:12:31.330529 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:31.330982 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:31.337348 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:31.337955 19423 maintenance_manager.cc:419] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Scheduling FlushMRSOp(10aa2feeb0324aa4a6221d2f2f3fc941): perf score=0.033364
I20240501 14:12:31.343770 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:31.343919 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:12:31.344429 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 14:12:31.352674 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:31.362095 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:31.376770 19685 maintenance_manager.cc:419] P ec8cde17a00645e7b38d5d29f7425367: Scheduling FlushMRSOp(fe554c3145de46cf8070efcecffbaedb): perf score=0.033374
I20240501 14:12:31.391714 19313 maintenance_manager.cc:643] P fb1e6bc2d07b4ce3b622a66f14c93bdc: FlushMRSOp(10aa2feeb0324aa4a6221d2f2f3fc941) complete. Timing: real 0.053s user 0.013s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":4073,"dirs.run_cpu_time_us":203,"dirs.run_wall_time_us":13148,"drs_written":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4,"lbm_write_time_us":665,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":137,"threads_started":2,"wal-append.queue_time_us":13449}
I20240501 14:12:31.392874 19423 maintenance_manager.cc:419] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Scheduling UndoDeltaBlockGCOp(10aa2feeb0324aa4a6221d2f2f3fc941): 4834 bytes on disk
I20240501 14:12:31.393610 19313 maintenance_manager.cc:643] P fb1e6bc2d07b4ce3b622a66f14c93bdc: UndoDeltaBlockGCOp(10aa2feeb0324aa4a6221d2f2f3fc941) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4}
I20240501 14:12:31.397754 19575 maintenance_manager.cc:643] P ec8cde17a00645e7b38d5d29f7425367: FlushMRSOp(fe554c3145de46cf8070efcecffbaedb) complete. Timing: real 0.021s user 0.018s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":382,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":623,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":154,"threads_started":2,"wal-append.queue_time_us":143}
I20240501 14:12:31.398908 19685 maintenance_manager.cc:419] P ec8cde17a00645e7b38d5d29f7425367: Scheduling UndoDeltaBlockGCOp(fe554c3145de46cf8070efcecffbaedb): 4921 bytes on disk
I20240501 14:12:31.403445 19575 maintenance_manager.cc:643] P ec8cde17a00645e7b38d5d29f7425367: UndoDeltaBlockGCOp(fe554c3145de46cf8070efcecffbaedb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240501 14:12:31.405352 19685 maintenance_manager.cc:419] P ec8cde17a00645e7b38d5d29f7425367: Scheduling FlushMRSOp(10aa2feeb0324aa4a6221d2f2f3fc941): perf score=0.033380
I20240501 14:12:31.413532 19816 maintenance_manager.cc:419] P da3c4e948ff747678f18456c134d7ed4: Scheduling FlushMRSOp(7eea58d4a33f422a9ada1e82d408577f): perf score=0.033384
I20240501 14:12:31.430941 19575 maintenance_manager.cc:643] P ec8cde17a00645e7b38d5d29f7425367: FlushMRSOp(10aa2feeb0324aa4a6221d2f2f3fc941) complete. Timing: real 0.025s user 0.019s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":46,"dirs.run_cpu_time_us":158,"dirs.run_wall_time_us":1294,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":637,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":73,"threads_started":1,"wal-append.queue_time_us":3762}
I20240501 14:12:31.432021 19685 maintenance_manager.cc:419] P ec8cde17a00645e7b38d5d29f7425367: Scheduling UndoDeltaBlockGCOp(10aa2feeb0324aa4a6221d2f2f3fc941): 4834 bytes on disk
I20240501 14:12:31.432621 19575 maintenance_manager.cc:643] P ec8cde17a00645e7b38d5d29f7425367: UndoDeltaBlockGCOp(10aa2feeb0324aa4a6221d2f2f3fc941) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
I20240501 14:12:31.435500 19706 maintenance_manager.cc:643] P da3c4e948ff747678f18456c134d7ed4: FlushMRSOp(7eea58d4a33f422a9ada1e82d408577f) complete. Timing: real 0.021s user 0.016s sys 0.001s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":189,"dirs.run_cpu_time_us":192,"dirs.run_wall_time_us":2841,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":543,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":163,"threads_started":2,"wal-append.queue_time_us":168}
I20240501 14:12:31.436451 19816 maintenance_manager.cc:419] P da3c4e948ff747678f18456c134d7ed4: Scheduling UndoDeltaBlockGCOp(7eea58d4a33f422a9ada1e82d408577f): 4581 bytes on disk
I20240501 14:12:31.436781 19706 maintenance_manager.cc:643] P da3c4e948ff747678f18456c134d7ed4: UndoDeltaBlockGCOp(7eea58d4a33f422a9ada1e82d408577f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4}
W20240501 14:12:31.460377 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:12:31.467801 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 14:12:31.489022 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:31.499290 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:31.504477 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:31.508658 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:31.514411 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:31.523197 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:31.523669 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:31.525257 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:31.530761 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:31.566114 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:31.566202 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:31.572235 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:31.573151 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:31.573185 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:31.604411 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 14:12:31.605121 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 14:12:31.675665 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:31.681607 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:31.682726 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:31.682914 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:31.684684 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:31.688759 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:31.712823 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:31.720793 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 14:12:31.806690 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:31.806854 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:31.811115 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:31.813540 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:31.815431 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:31.835634 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:31.837531 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:31.860862 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 14:12:31.926147 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:31.927644 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:31.928169 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:31.931197 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:31.937140 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:31.942075 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:31.942909 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:31.970639 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:32.008128 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:32.008210 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:32.011067 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:32.014717 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:32.014783 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:32.025171 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:32.036653 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:32.087677 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 14:12:32.088627 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:32.093096 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:32.099215 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:32.107308 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:32.111279 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:32.112797 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:32.119318 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:32.152740 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 14:12:32.176259 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:32.178906 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:32.182854 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:32.187981 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:32.196295 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:32.199023 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:32.208175 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:32.262476 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 14:12:32.302119 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 14:12:32.425734 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:32.426273 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:32.435034 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:32.438280 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:32.447270 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:32.450251 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:32.451021 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:32.456056 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 14:12:32.482677 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:12:32.548702 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 14:12:32.565764 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:32.566530 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:32.570271 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:32.574381 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:32.593516 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:32.601846 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:32.601826 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:32.627429 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:32.628494 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:32.633036 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:32.639496 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:32.641216 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:32.649974 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:32.650381 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:32.748241 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:32.751351 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:32.753497 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:32.759505 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:32.760941 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:32.781939 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:32.785995 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:32.833492 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 14:12:32.872197 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:32.872766 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 14:12:32.874608 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:32.876238 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:32.877568 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:32.877727 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:32.878743 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:32.888338 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:32.968549 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 14:12:32.986343 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:32.986591 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:32.989786 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:32.996649 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:33.017887 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:33.019258 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:33.027452 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:33.047120 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:33.051098 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:33.064730 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:33.068444 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:33.069042 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:33.070204 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:33.079821 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:33.080554 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:33.185252 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:33.192309 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:33.195631 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:33.196096 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:33.197875 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:33.210592 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:33.216411 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:33.222951 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:33.228775 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:33.233070 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:33.236369 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 14:12:33.237162 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:33.250561 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:33.264686 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:33.272286 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:12:33.282205 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:12:33.324441 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:12:33.414837 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:33.558967 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:33.568091 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:33.569496 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:33.570135 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:33.570271 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:33.579444 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:33.583463 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:33.620677 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:33.623070 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:33.632284 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:33.647614 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:33.651258 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:33.655944 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:33.656625 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:33.696633 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 14:12:33.757651 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 14:12:33.771159 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:33.777724 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:33.778319 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:33.783099 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:33.785974 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:33.789227 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:33.823801 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:33.827693 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:33.832094 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:33.842058 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:33.846418 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 14:12:33.849825 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:33.851637 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:33.863096 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:33.887928 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:33.958931 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:33.974555 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:33.977169 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:33.979429 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:33.980357 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:33.983558 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:33.995671 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:34.004139 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:34.030650 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:12:34.052453 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 14:12:34.108386 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:34.108714 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:34.110196 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:34.111980 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:34.113914 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:34.119643 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:34.130604 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:34.227830 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:34.230504 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:34.236294 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:34.237680 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:34.241494 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:34.243521 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:34.244889 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:34.313449 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:34.318842 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:34.321957 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:34.322147 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:34.326632 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:34.333881 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:34.338178 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:34.405974 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:34.406729 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:34.408027 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:34.415645 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:34.420516 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:34.432160 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:34.440294 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:34.448757 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:12:34.551275 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:34.649868 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:34.657999 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:34.660877 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:34.662338 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:34.667495 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:34.667963 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:34.689092 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:34.741183 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 14:12:34.777763 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:34.779444 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:34.790027 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 14:12:34.795176 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:34.796168 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:34.797868 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:34.799211 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 14:12:34.801507 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:34.833037 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:34.849013 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:34.851907 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:34.853219 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:34.853752 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:34.860097 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:34.880074 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:34.887528 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:34.906092 21446 maintenance_manager.cc:419] P 43ce4a4c954f4a5f80954b38933c189d: Scheduling FlushMRSOp(b6be9a115fe14c098c29cad819849f95): perf score=0.033336
I20240501 14:12:34.942394 21336 maintenance_manager.cc:643] P 43ce4a4c954f4a5f80954b38933c189d: FlushMRSOp(b6be9a115fe14c098c29cad819849f95) complete. Timing: real 0.036s user 0.015s sys 0.016s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":163,"dirs.queue_time_us":180,"dirs.run_cpu_time_us":207,"dirs.run_wall_time_us":6539,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":815,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":277,"threads_started":3,"wal-append.queue_time_us":176}
I20240501 14:12:34.944063 21446 maintenance_manager.cc:419] P 43ce4a4c954f4a5f80954b38933c189d: Scheduling UndoDeltaBlockGCOp(b6be9a115fe14c098c29cad819849f95): 4581 bytes on disk
I20240501 14:12:34.945006 21336 maintenance_manager.cc:643] P 43ce4a4c954f4a5f80954b38933c189d: UndoDeltaBlockGCOp(b6be9a115fe14c098c29cad819849f95) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
I20240501 14:12:34.990906 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:34.994956 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:35.000630 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:35.004149 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:35.021011 21577 maintenance_manager.cc:419] P 778c78105092428bbcde75ab2cdb6a05: Scheduling FlushMRSOp(cf62635e9a994259b9e293e5eb425b55): perf score=0.033378
I20240501 14:12:35.026680 21184 maintenance_manager.cc:419] P 5bf81f3d986e428190d3f7727f36d96c: Scheduling FlushMRSOp(0988c468757d49738765bdcb31794bd1): perf score=0.033334
W20240501 14:12:35.034091 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 14:12:35.036083 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:35.045948 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:35.046830 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:35.053711 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:35.124624 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:35.133953 21467 maintenance_manager.cc:643] P 778c78105092428bbcde75ab2cdb6a05: FlushMRSOp(cf62635e9a994259b9e293e5eb425b55) complete. Timing: real 0.112s user 0.023s sys 0.009s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":1649,"dirs.queue_time_us":7972,"dirs.run_cpu_time_us":122,"dirs.run_wall_time_us":1107,"drs_written":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4,"lbm_write_time_us":7173,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":239,"threads_started":3,"wal-append.queue_time_us":5282}
I20240501 14:12:35.135772 21577 maintenance_manager.cc:419] P 778c78105092428bbcde75ab2cdb6a05: Scheduling UndoDeltaBlockGCOp(cf62635e9a994259b9e293e5eb425b55): 4834 bytes on disk
I20240501 14:12:35.138078 21074 maintenance_manager.cc:643] P 5bf81f3d986e428190d3f7727f36d96c: FlushMRSOp(0988c468757d49738765bdcb31794bd1) complete. Timing: real 0.111s user 0.028s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":34678,"dirs.queue_time_us":8311,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":874,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":608,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":254,"threads_started":3,"wal-append.queue_time_us":8786}
I20240501 14:12:35.139451 21467 maintenance_manager.cc:643] P 778c78105092428bbcde75ab2cdb6a05: UndoDeltaBlockGCOp(cf62635e9a994259b9e293e5eb425b55) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4}
I20240501 14:12:35.139673 21184 maintenance_manager.cc:419] P 5bf81f3d986e428190d3f7727f36d96c: Scheduling UndoDeltaBlockGCOp(0988c468757d49738765bdcb31794bd1): 4581 bytes on disk
I20240501 14:12:35.140218 21074 maintenance_manager.cc:643] P 5bf81f3d986e428190d3f7727f36d96c: UndoDeltaBlockGCOp(0988c468757d49738765bdcb31794bd1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
I20240501 14:12:35.141045 21184 maintenance_manager.cc:419] P 5bf81f3d986e428190d3f7727f36d96c: Scheduling FlushMRSOp(7eea58d4a33f422a9ada1e82d408577f): perf score=0.033362
I20240501 14:12:35.143808 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:35.144886 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:35.145390 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:35.150934 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:35.177349 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:35.183653 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:35.196862 21446 maintenance_manager.cc:419] P 43ce4a4c954f4a5f80954b38933c189d: Scheduling FlushMRSOp(ad7c255baee74e629e2976a942a4174f): perf score=0.033401
I20240501 14:12:35.244870 21336 maintenance_manager.cc:643] P 43ce4a4c954f4a5f80954b38933c189d: FlushMRSOp(ad7c255baee74e629e2976a942a4174f) complete. Timing: real 0.048s user 0.019s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":43,"dirs.run_cpu_time_us":164,"dirs.run_wall_time_us":1243,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":572,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":103,"threads_started":1,"wal-append.queue_time_us":7491}
I20240501 14:12:35.246287 21446 maintenance_manager.cc:419] P 43ce4a4c954f4a5f80954b38933c189d: Scheduling FlushMRSOp(98af8b06b9324e3fa7cdb898b7b9fab9): perf score=0.033403
I20240501 14:12:35.250198 21074 maintenance_manager.cc:643] P 5bf81f3d986e428190d3f7727f36d96c: FlushMRSOp(7eea58d4a33f422a9ada1e82d408577f) complete. Timing: real 0.106s user 0.021s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":43,"dirs.run_cpu_time_us":221,"dirs.run_wall_time_us":1397,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":637,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":119,"threads_started":1,"wal-append.queue_time_us":5883}
I20240501 14:12:35.251230 21184 maintenance_manager.cc:419] P 5bf81f3d986e428190d3f7727f36d96c: Scheduling UndoDeltaBlockGCOp(7eea58d4a33f422a9ada1e82d408577f): 4581 bytes on disk
I20240501 14:12:35.251708 21074 maintenance_manager.cc:643] P 5bf81f3d986e428190d3f7727f36d96c: UndoDeltaBlockGCOp(7eea58d4a33f422a9ada1e82d408577f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
I20240501 14:12:35.252435 21184 maintenance_manager.cc:419] P 5bf81f3d986e428190d3f7727f36d96c: Scheduling FlushMRSOp(f69020bb6c244848a6b0a50fd2b4f3eb): perf score=0.033364
I20240501 14:12:35.290361 21336 maintenance_manager.cc:643] P 43ce4a4c954f4a5f80954b38933c189d: FlushMRSOp(98af8b06b9324e3fa7cdb898b7b9fab9) complete. Timing: real 0.044s user 0.020s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":42,"dirs.run_cpu_time_us":162,"dirs.run_wall_time_us":1224,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":708,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":109,"threads_started":1,"wal-append.queue_time_us":8086}
I20240501 14:12:35.291849 21446 maintenance_manager.cc:419] P 43ce4a4c954f4a5f80954b38933c189d: Scheduling UndoDeltaBlockGCOp(98af8b06b9324e3fa7cdb898b7b9fab9): 4834 bytes on disk
I20240501 14:12:35.292335 21336 maintenance_manager.cc:643] P 43ce4a4c954f4a5f80954b38933c189d: UndoDeltaBlockGCOp(98af8b06b9324e3fa7cdb898b7b9fab9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20240501 14:12:35.293431 21446 maintenance_manager.cc:419] P 43ce4a4c954f4a5f80954b38933c189d: Scheduling FlushMRSOp(7eea58d4a33f422a9ada1e82d408577f): perf score=0.033413
I20240501 14:12:35.311311 21074 maintenance_manager.cc:643] P 5bf81f3d986e428190d3f7727f36d96c: FlushMRSOp(f69020bb6c244848a6b0a50fd2b4f3eb) complete. Timing: real 0.059s user 0.024s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":43,"dirs.run_cpu_time_us":177,"dirs.run_wall_time_us":1467,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":661,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":143,"threads_started":1,"wal-append.queue_time_us":14284}
I20240501 14:12:35.312404 21184 maintenance_manager.cc:419] P 5bf81f3d986e428190d3f7727f36d96c: Scheduling UndoDeltaBlockGCOp(f69020bb6c244848a6b0a50fd2b4f3eb): 4921 bytes on disk
I20240501 14:12:35.312829 21074 maintenance_manager.cc:643] P 5bf81f3d986e428190d3f7727f36d96c: UndoDeltaBlockGCOp(f69020bb6c244848a6b0a50fd2b4f3eb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
I20240501 14:12:35.326462 21336 maintenance_manager.cc:643] P 43ce4a4c954f4a5f80954b38933c189d: FlushMRSOp(7eea58d4a33f422a9ada1e82d408577f) complete. Timing: real 0.033s user 0.019s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":28087,"compiler_manager_pool.run_cpu_time_us":29,"compiler_manager_pool.run_wall_time_us":30,"dirs.queue_time_us":38,"dirs.run_cpu_time_us":187,"dirs.run_wall_time_us":1292,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3704,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":114,"threads_started":1,"wal-append.queue_time_us":5666}
I20240501 14:12:35.327841 21446 maintenance_manager.cc:419] P 43ce4a4c954f4a5f80954b38933c189d: Scheduling FlushMRSOp(db0f01ee72b24ec9b84410cf94ef9759): perf score=0.033405
W20240501 14:12:35.334467 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:12:35.335270 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 14:12:35.356846 21336 maintenance_manager.cc:643] P 43ce4a4c954f4a5f80954b38933c189d: FlushMRSOp(db0f01ee72b24ec9b84410cf94ef9759) complete. Timing: real 0.029s user 0.015s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":264,"dirs.run_cpu_time_us":168,"dirs.run_wall_time_us":1469,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":660,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":108,"threads_started":1,"wal-append.queue_time_us":8356}
I20240501 14:12:35.357856 21446 maintenance_manager.cc:419] P 43ce4a4c954f4a5f80954b38933c189d: Scheduling UndoDeltaBlockGCOp(ad7c255baee74e629e2976a942a4174f): 4834 bytes on disk
I20240501 14:12:35.358275 21336 maintenance_manager.cc:643] P 43ce4a4c954f4a5f80954b38933c189d: UndoDeltaBlockGCOp(ad7c255baee74e629e2976a942a4174f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
I20240501 14:12:35.359032 21446 maintenance_manager.cc:419] P 43ce4a4c954f4a5f80954b38933c189d: Scheduling UndoDeltaBlockGCOp(7eea58d4a33f422a9ada1e82d408577f): 4581 bytes on disk
I20240501 14:12:35.359697 21336 maintenance_manager.cc:643] P 43ce4a4c954f4a5f80954b38933c189d: UndoDeltaBlockGCOp(7eea58d4a33f422a9ada1e82d408577f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20240501 14:12:35.360399 21446 maintenance_manager.cc:419] P 43ce4a4c954f4a5f80954b38933c189d: Scheduling UndoDeltaBlockGCOp(db0f01ee72b24ec9b84410cf94ef9759): 4581 bytes on disk
I20240501 14:12:35.360733 21336 maintenance_manager.cc:643] P 43ce4a4c954f4a5f80954b38933c189d: UndoDeltaBlockGCOp(db0f01ee72b24ec9b84410cf94ef9759) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240501 14:12:35.386368 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:35.388371 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:35.391283 21577 maintenance_manager.cc:419] P 778c78105092428bbcde75ab2cdb6a05: Scheduling FlushMRSOp(a9edf1764b55420ea5d519e07ae1d5e0): perf score=0.033401
I20240501 14:12:35.393738 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:35.397735 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:35.399874 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:35.414503 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:35.422348 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:35.438918 21467 maintenance_manager.cc:643] P 778c78105092428bbcde75ab2cdb6a05: FlushMRSOp(a9edf1764b55420ea5d519e07ae1d5e0) complete. Timing: real 0.047s user 0.011s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":87,"dirs.run_cpu_time_us":175,"dirs.run_wall_time_us":1609,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":618,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":108,"threads_started":1,"wal-append.queue_time_us":31060}
I20240501 14:12:35.439966 21577 maintenance_manager.cc:419] P 778c78105092428bbcde75ab2cdb6a05: Scheduling FlushMRSOp(55830424a98348e382759c54afdeabcd): perf score=0.033335
I20240501 14:12:35.458626 21467 maintenance_manager.cc:643] P 778c78105092428bbcde75ab2cdb6a05: FlushMRSOp(55830424a98348e382759c54afdeabcd) complete. Timing: real 0.018s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":38,"dirs.run_cpu_time_us":208,"dirs.run_wall_time_us":1446,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":555,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":97,"threads_started":1,"wal-append.queue_time_us":166}
I20240501 14:12:35.459635 21577 maintenance_manager.cc:419] P 778c78105092428bbcde75ab2cdb6a05: Scheduling UndoDeltaBlockGCOp(55830424a98348e382759c54afdeabcd): 4581 bytes on disk
I20240501 14:12:35.460012 21467 maintenance_manager.cc:643] P 778c78105092428bbcde75ab2cdb6a05: UndoDeltaBlockGCOp(55830424a98348e382759c54afdeabcd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20240501 14:12:35.460688 21577 maintenance_manager.cc:419] P 778c78105092428bbcde75ab2cdb6a05: Scheduling UndoDeltaBlockGCOp(a9edf1764b55420ea5d519e07ae1d5e0): 4581 bytes on disk
I20240501 14:12:35.461036 21467 maintenance_manager.cc:643] P 778c78105092428bbcde75ab2cdb6a05: UndoDeltaBlockGCOp(a9edf1764b55420ea5d519e07ae1d5e0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
W20240501 14:12:35.471954 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 14:12:35.513420 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:35.517066 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:35.524019 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:35.533172 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:35.539249 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:35.542243 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:35.571475 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:35.572027 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:35.591120 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 14:12:35.612226 21446 maintenance_manager.cc:419] P 43ce4a4c954f4a5f80954b38933c189d: Scheduling FlushMRSOp(10aa2feeb0324aa4a6221d2f2f3fc941): perf score=0.033375
I20240501 14:12:35.623342 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:35.631596 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:35.633358 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:35.634619 21336 maintenance_manager.cc:643] P 43ce4a4c954f4a5f80954b38933c189d: FlushMRSOp(10aa2feeb0324aa4a6221d2f2f3fc941) complete. Timing: real 0.022s user 0.014s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":3406,"dirs.run_cpu_time_us":170,"dirs.run_wall_time_us":1480,"drs_written":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4,"lbm_write_time_us":527,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":65,"threads_started":1,"wal-append.queue_time_us":6539}
I20240501 14:12:35.635591 21446 maintenance_manager.cc:419] P 43ce4a4c954f4a5f80954b38933c189d: Scheduling UndoDeltaBlockGCOp(10aa2feeb0324aa4a6221d2f2f3fc941): 4834 bytes on disk
I20240501 14:12:35.635962 21336 maintenance_manager.cc:643] P 43ce4a4c954f4a5f80954b38933c189d: UndoDeltaBlockGCOp(10aa2feeb0324aa4a6221d2f2f3fc941) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240501 14:12:35.640360 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:35.645375 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:35.652807 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:35.657374 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:35.712492 21577 maintenance_manager.cc:419] P 778c78105092428bbcde75ab2cdb6a05: Scheduling FlushMRSOp(10aa2feeb0324aa4a6221d2f2f3fc941): perf score=0.033381
I20240501 14:12:35.721745 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:35.721894 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:35.721983 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:35.727216 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:35.731539 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:35.740876 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:35.755477 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 14:12:35.760007 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:35.765923 21467 maintenance_manager.cc:643] P 778c78105092428bbcde75ab2cdb6a05: FlushMRSOp(10aa2feeb0324aa4a6221d2f2f3fc941) complete. Timing: real 0.053s user 0.016s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":36,"dirs.run_cpu_time_us":148,"dirs.run_wall_time_us":1440,"drs_written":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4,"lbm_write_time_us":593,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":78,"threads_started":1,"wal-append.queue_time_us":4098}
I20240501 14:12:35.767699 21577 maintenance_manager.cc:419] P 778c78105092428bbcde75ab2cdb6a05: Scheduling UndoDeltaBlockGCOp(10aa2feeb0324aa4a6221d2f2f3fc941): 4834 bytes on disk
I20240501 14:12:35.768096 21467 maintenance_manager.cc:643] P 778c78105092428bbcde75ab2cdb6a05: UndoDeltaBlockGCOp(10aa2feeb0324aa4a6221d2f2f3fc941) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240501 14:12:35.815472 21184 maintenance_manager.cc:419] P 5bf81f3d986e428190d3f7727f36d96c: Scheduling FlushMRSOp(98af8b06b9324e3fa7cdb898b7b9fab9): perf score=0.033396
I20240501 14:12:35.827633 21074 maintenance_manager.cc:643] P 5bf81f3d986e428190d3f7727f36d96c: FlushMRSOp(98af8b06b9324e3fa7cdb898b7b9fab9) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":249,"dirs.run_cpu_time_us":243,"dirs.run_wall_time_us":1566,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":490,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":157,"threads_started":2,"wal-append.queue_time_us":152}
I20240501 14:12:35.828397 21184 maintenance_manager.cc:419] P 5bf81f3d986e428190d3f7727f36d96c: Scheduling FlushMRSOp(ec2a46f8cfb3406082970e5fdb59b101): perf score=0.033387
W20240501 14:12:35.835947 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 14:12:35.840399 21074 maintenance_manager.cc:643] P 5bf81f3d986e428190d3f7727f36d96c: FlushMRSOp(ec2a46f8cfb3406082970e5fdb59b101) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":45,"dirs.run_cpu_time_us":149,"dirs.run_wall_time_us":1294,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":396,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":147,"threads_started":1,"wal-append.queue_time_us":223}
I20240501 14:12:35.841079 21184 maintenance_manager.cc:419] P 5bf81f3d986e428190d3f7727f36d96c: Scheduling UndoDeltaBlockGCOp(98af8b06b9324e3fa7cdb898b7b9fab9): 4834 bytes on disk
I20240501 14:12:35.841413 21074 maintenance_manager.cc:643] P 5bf81f3d986e428190d3f7727f36d96c: UndoDeltaBlockGCOp(98af8b06b9324e3fa7cdb898b7b9fab9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4}
I20240501 14:12:35.842010 21184 maintenance_manager.cc:419] P 5bf81f3d986e428190d3f7727f36d96c: Scheduling UndoDeltaBlockGCOp(ec2a46f8cfb3406082970e5fdb59b101): 4832 bytes on disk
I20240501 14:12:35.842293 21074 maintenance_manager.cc:643] P 5bf81f3d986e428190d3f7727f36d96c: UndoDeltaBlockGCOp(ec2a46f8cfb3406082970e5fdb59b101) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":42,"lbm_reads_lt_1ms":4}
I20240501 14:12:35.963361 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:35.966564 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:35.970162 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:35.970641 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:35.981175 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:36.001914 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:36.008152 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:36.023907 21577 maintenance_manager.cc:419] P 778c78105092428bbcde75ab2cdb6a05: Scheduling FlushMRSOp(272eee3450cf4f48a4e1de2dcb28c0f7): perf score=0.033402
W20240501 14:12:36.027820 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:36.049132 21467 maintenance_manager.cc:643] P 778c78105092428bbcde75ab2cdb6a05: FlushMRSOp(272eee3450cf4f48a4e1de2dcb28c0f7) complete. Timing: real 0.022s user 0.017s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":52,"dirs.run_cpu_time_us":166,"dirs.run_wall_time_us":1606,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":536,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":91,"threads_started":1,"wal-append.queue_time_us":2503}
I20240501 14:12:36.050118 21577 maintenance_manager.cc:419] P 778c78105092428bbcde75ab2cdb6a05: Scheduling UndoDeltaBlockGCOp(272eee3450cf4f48a4e1de2dcb28c0f7): 4834 bytes on disk
I20240501 14:12:36.050647 21467 maintenance_manager.cc:643] P 778c78105092428bbcde75ab2cdb6a05: UndoDeltaBlockGCOp(272eee3450cf4f48a4e1de2dcb28c0f7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240501 14:12:36.051517 21577 maintenance_manager.cc:419] P 778c78105092428bbcde75ab2cdb6a05: Scheduling FlushMRSOp(410a7dd0dd734684a82875912966657d): perf score=0.033394
I20240501 14:12:36.068631 21467 maintenance_manager.cc:643] P 778c78105092428bbcde75ab2cdb6a05: FlushMRSOp(410a7dd0dd734684a82875912966657d) complete. Timing: real 0.017s user 0.010s sys 0.005s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":54,"dirs.run_cpu_time_us":166,"dirs.run_wall_time_us":1334,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":501,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":116,"threads_started":1,"wal-append.queue_time_us":210}
I20240501 14:12:36.069464 21577 maintenance_manager.cc:419] P 778c78105092428bbcde75ab2cdb6a05: Scheduling UndoDeltaBlockGCOp(410a7dd0dd734684a82875912966657d): 4832 bytes on disk
I20240501 14:12:36.071166 21467 maintenance_manager.cc:643] P 778c78105092428bbcde75ab2cdb6a05: UndoDeltaBlockGCOp(410a7dd0dd734684a82875912966657d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240501 14:12:36.093516 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:36.096890 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:36.099797 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:36.099877 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:36.112259 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:36.113029 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:36.125006 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:36.167773 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:36.168314 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:36.175153 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:36.177497 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:36.179729 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:36.189049 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:36.209718 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:36.247239 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 14:12:36.277458 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:36.278497 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:36.285862 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:36.286865 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:36.287894 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:36.306639 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:36.309988 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:36.344831 21184 maintenance_manager.cc:419] P 5bf81f3d986e428190d3f7727f36d96c: Scheduling FlushMRSOp(1d408b075d784953a57554bcdf19158a): perf score=0.033394
I20240501 14:12:36.355639 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:36.359145 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:36.359325 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:36.366077 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:36.366812 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:36.390376 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:36.399140 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:36.399981 21074 maintenance_manager.cc:643] P 5bf81f3d986e428190d3f7727f36d96c: FlushMRSOp(1d408b075d784953a57554bcdf19158a) complete. Timing: real 0.055s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":31939,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":578,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":32730,"threads_started":2,"wal-append.queue_time_us":32576}
I20240501 14:12:36.400971 21184 maintenance_manager.cc:419] P 5bf81f3d986e428190d3f7727f36d96c: Scheduling UndoDeltaBlockGCOp(1d408b075d784953a57554bcdf19158a): 4921 bytes on disk
I20240501 14:12:36.404413 21074 maintenance_manager.cc:643] P 5bf81f3d986e428190d3f7727f36d96c: UndoDeltaBlockGCOp(1d408b075d784953a57554bcdf19158a) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
W20240501 14:12:36.409438 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:12:36.477881 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 14:12:36.495587 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:36.496819 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:36.501742 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 14:12:36.503736 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:36.503974 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:36.504097 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:36.520164 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:36.520625 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:36.573761 21577 maintenance_manager.cc:419] P 778c78105092428bbcde75ab2cdb6a05: Scheduling FlushMRSOp(ec2a46f8cfb3406082970e5fdb59b101): perf score=0.033373
I20240501 14:12:36.594795 21467 maintenance_manager.cc:643] P 778c78105092428bbcde75ab2cdb6a05: FlushMRSOp(ec2a46f8cfb3406082970e5fdb59b101) complete. Timing: real 0.021s user 0.018s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":4717,"dirs.run_cpu_time_us":154,"dirs.run_wall_time_us":1678,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":736,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":141,"threads_started":2,"wal-append.queue_time_us":266}
I20240501 14:12:36.595849 21577 maintenance_manager.cc:419] P 778c78105092428bbcde75ab2cdb6a05: Scheduling UndoDeltaBlockGCOp(ec2a46f8cfb3406082970e5fdb59b101): 4832 bytes on disk
I20240501 14:12:36.596621 21467 maintenance_manager.cc:643] P 778c78105092428bbcde75ab2cdb6a05: UndoDeltaBlockGCOp(ec2a46f8cfb3406082970e5fdb59b101) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240501 14:12:36.597766 21577 maintenance_manager.cc:419] P 778c78105092428bbcde75ab2cdb6a05: Scheduling FlushMRSOp(bb291a0c743c4e638f79da36cc77faa3): perf score=0.033336
W20240501 14:12:36.604959 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 14:12:36.619494 21467 maintenance_manager.cc:643] P 778c78105092428bbcde75ab2cdb6a05: FlushMRSOp(bb291a0c743c4e638f79da36cc77faa3) complete. Timing: real 0.021s user 0.016s sys 0.001s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":40,"dirs.run_cpu_time_us":200,"dirs.run_wall_time_us":1486,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":669,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":72,"threads_started":1,"wal-append.queue_time_us":143}
I20240501 14:12:36.620411 21577 maintenance_manager.cc:419] P 778c78105092428bbcde75ab2cdb6a05: Scheduling UndoDeltaBlockGCOp(bb291a0c743c4e638f79da36cc77faa3): 4581 bytes on disk
I20240501 14:12:36.620829 21467 maintenance_manager.cc:643] P 778c78105092428bbcde75ab2cdb6a05: UndoDeltaBlockGCOp(bb291a0c743c4e638f79da36cc77faa3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20240501 14:12:36.630818 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:36.632109 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:36.638271 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:36.639524 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:36.639586 21446 maintenance_manager.cc:419] P 43ce4a4c954f4a5f80954b38933c189d: Scheduling FlushMRSOp(2dcca6e0e6374520bd4522c80c110e97): perf score=0.033390
I20240501 14:12:36.646040 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:36.649431 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:36.655828 21184 maintenance_manager.cc:419] P 5bf81f3d986e428190d3f7727f36d96c: Scheduling FlushMRSOp(bb291a0c743c4e638f79da36cc77faa3): perf score=0.033375
I20240501 14:12:36.656123 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:36.666404 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:36.675498 21336 maintenance_manager.cc:643] P 43ce4a4c954f4a5f80954b38933c189d: FlushMRSOp(2dcca6e0e6374520bd4522c80c110e97) complete. Timing: real 0.035s user 0.017s sys 0.001s Metrics: {"bytes_written":6144,"cfile_init":1,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":575,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":158,"threads_started":2,"wal-append.queue_time_us":160}
I20240501 14:12:36.676354 21446 maintenance_manager.cc:419] P 43ce4a4c954f4a5f80954b38933c189d: Scheduling UndoDeltaBlockGCOp(2dcca6e0e6374520bd4522c80c110e97): 4920 bytes on disk
I20240501 14:12:36.676802 21336 maintenance_manager.cc:643] P 43ce4a4c954f4a5f80954b38933c189d: UndoDeltaBlockGCOp(2dcca6e0e6374520bd4522c80c110e97) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240501 14:12:36.677515 21446 maintenance_manager.cc:419] P 43ce4a4c954f4a5f80954b38933c189d: Scheduling FlushMRSOp(cc8e888d46d34a22b6d4409ec2ab53a9): perf score=0.033399
I20240501 14:12:36.681095 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:36.694437 21336 maintenance_manager.cc:643] P 43ce4a4c954f4a5f80954b38933c189d: FlushMRSOp(cc8e888d46d34a22b6d4409ec2ab53a9) complete. Timing: real 0.017s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":601,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":231,"threads_started":2,"wal-append.queue_time_us":292}
I20240501 14:12:36.695499 21446 maintenance_manager.cc:419] P 43ce4a4c954f4a5f80954b38933c189d: Scheduling FlushMRSOp(f6ca088ce3aa4b97aa54c8ce6d700921): perf score=0.033388
I20240501 14:12:36.717043 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:36.718101 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:36.718295 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:36.719238 21074 maintenance_manager.cc:643] P 5bf81f3d986e428190d3f7727f36d96c: FlushMRSOp(bb291a0c743c4e638f79da36cc77faa3) complete. Timing: real 0.063s user 0.016s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":9101,"dirs.run_cpu_time_us":216,"dirs.run_wall_time_us":1444,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":599,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":77,"threads_started":1,"wal-append.queue_time_us":171}
I20240501 14:12:36.720198 21184 maintenance_manager.cc:419] P 5bf81f3d986e428190d3f7727f36d96c: Scheduling UndoDeltaBlockGCOp(bb291a0c743c4e638f79da36cc77faa3): 4581 bytes on disk
I20240501 14:12:36.720551 21074 maintenance_manager.cc:643] P 5bf81f3d986e428190d3f7727f36d96c: UndoDeltaBlockGCOp(bb291a0c743c4e638f79da36cc77faa3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240501 14:12:36.722927 21336 maintenance_manager.cc:643] P 43ce4a4c954f4a5f80954b38933c189d: FlushMRSOp(f6ca088ce3aa4b97aa54c8ce6d700921) complete. Timing: real 0.027s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":5292,"dirs.run_cpu_time_us":136,"dirs.run_wall_time_us":1568,"drs_written":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4,"lbm_write_time_us":529,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":134,"threads_started":1,"wal-append.queue_time_us":6004}
I20240501 14:12:36.727818 21446 maintenance_manager.cc:419] P 43ce4a4c954f4a5f80954b38933c189d: Scheduling UndoDeltaBlockGCOp(cc8e888d46d34a22b6d4409ec2ab53a9): 4921 bytes on disk
I20240501 14:12:36.728430 21336 maintenance_manager.cc:643] P 43ce4a4c954f4a5f80954b38933c189d: UndoDeltaBlockGCOp(cc8e888d46d34a22b6d4409ec2ab53a9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
I20240501 14:12:36.729115 21446 maintenance_manager.cc:419] P 43ce4a4c954f4a5f80954b38933c189d: Scheduling UndoDeltaBlockGCOp(f6ca088ce3aa4b97aa54c8ce6d700921): 4581 bytes on disk
I20240501 14:12:36.729499 21336 maintenance_manager.cc:643] P 43ce4a4c954f4a5f80954b38933c189d: UndoDeltaBlockGCOp(f6ca088ce3aa4b97aa54c8ce6d700921) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4}
I20240501 14:12:36.731549 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:36.735484 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:36.823820 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:36.825242 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:36.832290 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:36.832957 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:12:36.835768 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:36.843448 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:36.846886 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:36.864601 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:36.907573 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 14:12:36.997205 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:36.997548 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 14:12:36.999363 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:36.999459 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:37.007548 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:37.007753 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:37.021278 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:37.037787 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:37.123225 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:37.135547 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:37.136852 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:37.138370 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:37.140739 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:37.141614 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:37.142279 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:12:37.200888 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 14:12:37.209537 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:37.209525 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:37.223882 21184 maintenance_manager.cc:419] P 5bf81f3d986e428190d3f7727f36d96c: Scheduling FlushMRSOp(dab5e5722e2e4b0eaea73288d8dce265): perf score=0.033342
I20240501 14:12:37.226231 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:37.227728 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:37.230854 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:37.232009 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:37.240674 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 14:12:37.256848 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:37.266291 21074 maintenance_manager.cc:643] P 5bf81f3d986e428190d3f7727f36d96c: FlushMRSOp(dab5e5722e2e4b0eaea73288d8dce265) complete. Timing: real 0.042s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"drs_written":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4,"lbm_write_time_us":675,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":191,"threads_started":2,"wal-append.queue_time_us":179}
I20240501 14:12:37.267630 21184 maintenance_manager.cc:419] P 5bf81f3d986e428190d3f7727f36d96c: Scheduling UndoDeltaBlockGCOp(dab5e5722e2e4b0eaea73288d8dce265): 4921 bytes on disk
I20240501 14:12:37.268253 21074 maintenance_manager.cc:643] P 5bf81f3d986e428190d3f7727f36d96c: UndoDeltaBlockGCOp(dab5e5722e2e4b0eaea73288d8dce265) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
I20240501 14:12:37.269016 21184 maintenance_manager.cc:419] P 5bf81f3d986e428190d3f7727f36d96c: Scheduling FlushMRSOp(78e14dea8b6e4c0abd1c88dcd5b4e608): perf score=0.033354
I20240501 14:12:37.289734 21074 maintenance_manager.cc:643] P 5bf81f3d986e428190d3f7727f36d96c: FlushMRSOp(78e14dea8b6e4c0abd1c88dcd5b4e608) complete. Timing: real 0.020s user 0.016s sys 0.001s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":51,"dirs.run_cpu_time_us":174,"dirs.run_wall_time_us":2722,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":601,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":73,"threads_started":1,"wal-append.queue_time_us":168}
I20240501 14:12:37.290800 21184 maintenance_manager.cc:419] P 5bf81f3d986e428190d3f7727f36d96c: Scheduling UndoDeltaBlockGCOp(78e14dea8b6e4c0abd1c88dcd5b4e608): 4921 bytes on disk
I20240501 14:12:37.291244 21074 maintenance_manager.cc:643] P 5bf81f3d986e428190d3f7727f36d96c: UndoDeltaBlockGCOp(78e14dea8b6e4c0abd1c88dcd5b4e608) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
I20240501 14:12:37.315153 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:37.317409 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:37.324224 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:37.335673 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:37.337620 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:37.343729 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:37.348428 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:37.353415 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:37.374001 21577 maintenance_manager.cc:419] P 778c78105092428bbcde75ab2cdb6a05: Scheduling FlushMRSOp(dadab886b03540579d138cdb5b25e286): perf score=0.033394
I20240501 14:12:37.401423 21467 maintenance_manager.cc:643] P 778c78105092428bbcde75ab2cdb6a05: FlushMRSOp(dadab886b03540579d138cdb5b25e286) complete. Timing: real 0.026s user 0.018s sys 0.002s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":4101,"dirs.run_cpu_time_us":213,"dirs.run_wall_time_us":2040,"drs_written":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4,"lbm_write_time_us":709,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":176,"threads_started":2,"wal-append.queue_time_us":438}
I20240501 14:12:37.402825 21577 maintenance_manager.cc:419] P 778c78105092428bbcde75ab2cdb6a05: Scheduling UndoDeltaBlockGCOp(dadab886b03540579d138cdb5b25e286): 4834 bytes on disk
I20240501 14:12:37.414870 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:37.420804 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:37.420917 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:37.421049 21467 maintenance_manager.cc:643] P 778c78105092428bbcde75ab2cdb6a05: UndoDeltaBlockGCOp(dadab886b03540579d138cdb5b25e286) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
I20240501 14:12:37.421847 21577 maintenance_manager.cc:419] P 778c78105092428bbcde75ab2cdb6a05: Scheduling FlushMRSOp(fe554c3145de46cf8070efcecffbaedb): perf score=0.033392
I20240501 14:12:37.422552 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:37.433239 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:37.433830 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:12:37.448529 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 14:12:37.449021 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:37.471357 21467 maintenance_manager.cc:643] P 778c78105092428bbcde75ab2cdb6a05: FlushMRSOp(fe554c3145de46cf8070efcecffbaedb) complete. Timing: real 0.032s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":34,"dirs.run_cpu_time_us":191,"dirs.run_wall_time_us":8104,"drs_written":1,"lbm_read_time_us":42,"lbm_reads_lt_1ms":4,"lbm_write_time_us":599,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":68,"threads_started":1,"wal-append.queue_time_us":14724}
I20240501 14:12:37.472153 21577 maintenance_manager.cc:419] P 778c78105092428bbcde75ab2cdb6a05: Scheduling UndoDeltaBlockGCOp(fe554c3145de46cf8070efcecffbaedb): 4921 bytes on disk
I20240501 14:12:37.472522 21467 maintenance_manager.cc:643] P 778c78105092428bbcde75ab2cdb6a05: UndoDeltaBlockGCOp(fe554c3145de46cf8070efcecffbaedb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4}
W20240501 14:12:37.524824 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:12:37.531536 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:37.532490 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:37.533990 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:37.534252 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:37.537884 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:37.539713 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:37.549140 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:37.648602 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:37.649834 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:37.651873 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:37.655382 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:37.672966 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:37.681195 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:37.683403 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:37.717885 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 14:12:37.764843 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:37.765161 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:37.770779 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:37.771343 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:37.773964 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:37.781445 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:37.782725 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:37.799486 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:37.885093 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:37.885907 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:37.886818 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:37.894531 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:37.895985 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:12:37.906427 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 14:12:37.911336 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:37.918972 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:37.954530 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 14:12:37.999137 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:38.000218 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:38.006393 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:38.012235 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:12:38.012297 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:12:38.020010 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:38.029949 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:38.029979 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:38.034054 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:12:38.068557 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 14:12:38.123795 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:38.124645 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:38.129012 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:38.133136 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:38.135738 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:38.144860 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:38.145337 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:38.222858 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:38.236681 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:38.237982 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:38.242071 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:38.251000 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:38.258124 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:38.259160 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 14:12:38.263525 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:38.276250 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:38.307121 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:38.311861 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:38.312439 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:38.313480 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:38.328944 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:38.339577 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:38.355223 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:38.435573 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:38.439276 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:38.439816 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:38.441668 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:38.441771 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:38.456413 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:38.472604 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:38.505694 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:38.506228 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:38.509518 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:38.510711 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:38.511844 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:12:38.517663 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:38.518172 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:38.519776 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 14:12:38.523555 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:38.619184 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:38.619843 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:38.620175 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:38.627636 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:38.628254 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:38.628360 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:38.653059 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:38.771425 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:38.772284 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240501 14:12:38.772770 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:38.775002 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:38.778025 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:38.785446 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:38.794415 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:38.799558 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:38.805505 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:38.901698 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:38.901749 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:38.905465 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:38.911957 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:38.914794 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:38.929251 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:38.939004 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:38.939167 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:12:38.956063 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 14:12:39.000257 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:39.006457 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:39.007304 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:39.008809 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:39.021268 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:39.027922 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:39.028192 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:39.101466 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 14:12:39.121228 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:39.124563 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:39.126852 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:39.130609 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:39.131227 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:39.132151 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:39.139469 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:39.229949 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:39.261890 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:39.268900 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:39.269831 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:39.269986 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:39.272572 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:39.272763 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:39.290019 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:39.309357 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:39.312407 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:39.312490 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:39.317206 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:39.319118 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:39.334548 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:39.355412 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:39.451340 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:39.453437 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:39.460978 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:39.468394 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:39.478456 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:39.486382 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:39.491724 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:39.495718 21446 maintenance_manager.cc:419] P 43ce4a4c954f4a5f80954b38933c189d: Scheduling FlushMRSOp(f69020bb6c244848a6b0a50fd2b4f3eb): perf score=0.033348
I20240501 14:12:39.513759 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:39.522301 21336 maintenance_manager.cc:643] P 43ce4a4c954f4a5f80954b38933c189d: FlushMRSOp(f69020bb6c244848a6b0a50fd2b4f3eb) complete. Timing: real 0.024s user 0.015s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":630,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":171,"threads_started":2,"wal-append.queue_time_us":164}
I20240501 14:12:39.523226 21446 maintenance_manager.cc:419] P 43ce4a4c954f4a5f80954b38933c189d: Scheduling UndoDeltaBlockGCOp(f69020bb6c244848a6b0a50fd2b4f3eb): 4921 bytes on disk
I20240501 14:12:39.523661 21336 maintenance_manager.cc:643] P 43ce4a4c954f4a5f80954b38933c189d: UndoDeltaBlockGCOp(f69020bb6c244848a6b0a50fd2b4f3eb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
I20240501 14:12:39.526127 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:39.526443 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:39.529199 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:39.539036 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:39.551088 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:39.564211 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:39.603948 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 14:12:39.630235 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:39.634876 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:39.636077 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:39.637472 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:39.639375 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:39.642311 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:39.651762 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:39.720526 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:12:39.784468 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:12:39.809981 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:12:39.822698 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 14:12:39.857960 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:39.866410 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:39.866648 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:39.869892 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:39.873719 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:39.874897 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:39.892665 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:39.976012 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:39.979212 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:39.991897 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:39.994086 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:40.003240 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:40.003924 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:40.018141 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:40.022850 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:40.031589 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:40.043242 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:40.056264 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:40.064368 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:40.065479 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:40.067133 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:40.149684 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 14:12:40.166080 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:40.166494 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:40.167273 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:40.169335 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:40.183112 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:40.183208 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:40.184024 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:40.277015 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:40.277741 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:40.279072 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:40.281821 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:12:40.292124 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 14:12:40.297307 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:40.305640 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:40.311838 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:40.378094 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:40.378620 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:12:40.379460 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:12:40.379853 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 14:12:40.387550 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:40.389915 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:40.414024 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:40.417543 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:40.425194 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:40.425652 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:40.477087 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:40.477504 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:40.489672 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:40.489754 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:40.490235 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:40.494807 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:40.506412 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:40.520843 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:12:40.546468 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 14:12:40.559957 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:40.565011 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:40.565965 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:40.574406 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:40.586094 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:40.600369 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:40.603483 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:40.649513 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:40.651204 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:40.661331 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:40.661680 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:40.663332 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:40.682961 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:40.683359 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:40.816248 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:40.891940 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:40.898034 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:40.898952 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:40.900723 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:40.902364 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:40.909206 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:40.931314 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:40.992332 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:12:40.998447 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 14:12:41.025305 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:41.029948 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:41.035280 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:41.037482 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:41.037573 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:41.042945 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:41.055475 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:41.061218 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:41.061241 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:41.061527 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:41.067257 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:41.106164 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:41.108582 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:41.110270 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:41.194051 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:41.195904 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:41.197894 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:41.202036 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:41.208678 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:41.209357 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:41.224129 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:41.238355 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:41.305022 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:41.310595 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:41.311985 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:41.312387 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:41.313597 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:41.343822 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:41.344797 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:41.417745 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:41.429327 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 14:12:41.430546 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:41.432935 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:41.434316 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:41.434429 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:41.437790 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:12:41.440779 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 14:12:41.447650 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:41.513855 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:41.515436 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:41.516031 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:41.522387 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:41.523622 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:41.524315 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:41.534844 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 14:12:41.536309 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:41.609588 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:41.613070 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:41.614522 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:41.619504 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:41.634605 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:41.646128 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:41.651233 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:41.684433 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:41.687224 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:41.690466 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:41.699863 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:41.702634 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:41.703868 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:41.713032 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:41.784044 21446 maintenance_manager.cc:419] P 43ce4a4c954f4a5f80954b38933c189d: Scheduling FlushMRSOp(85593473500b4d5eaf9f8b0bf2296777): perf score=0.033376
I20240501 14:12:41.795608 21336 maintenance_manager.cc:643] P 43ce4a4c954f4a5f80954b38933c189d: FlushMRSOp(85593473500b4d5eaf9f8b0bf2296777) complete. Timing: real 0.011s user 0.011s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":177,"dirs.run_cpu_time_us":186,"dirs.run_wall_time_us":1875,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":398,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":147,"threads_started":2,"wal-append.queue_time_us":144}
I20240501 14:12:41.796308 21446 maintenance_manager.cc:419] P 43ce4a4c954f4a5f80954b38933c189d: Scheduling UndoDeltaBlockGCOp(85593473500b4d5eaf9f8b0bf2296777): 4834 bytes on disk
I20240501 14:12:41.796636 21336 maintenance_manager.cc:643] P 43ce4a4c954f4a5f80954b38933c189d: UndoDeltaBlockGCOp(85593473500b4d5eaf9f8b0bf2296777) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
W20240501 14:12:41.807484 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:41.939217 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:41.940310 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:41.941555 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:41.944177 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:41.951644 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:41.964785 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:41.974913 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:42.036931 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 14:12:42.060472 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 14:12:42.069474 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:42.072407 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:42.074319 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:42.080308 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:42.084380 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:42.092836 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:42.094573 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:42.099179 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:42.099681 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:42.099984 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:42.106364 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:42.111519 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:42.119307 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:42.148536 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 14:12:42.158067 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:42.241786 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:42.242369 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 14:12:42.244755 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:42.245244 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:42.246603 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:42.251478 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:42.256105 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:42.256361 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:42.279367 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:42.356004 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:42.358420 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:42.361181 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:42.370985 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:42.371588 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:42.382655 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:42.389614 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 14:12:42.396131 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:42.460107 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 14:12:42.494838 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:42.502511 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:42.502657 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:42.507000 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:42.513907 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:42.515394 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:42.525975 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:42.528671 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:42.531268 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:42.541824 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:42.553210 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:42.573225 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:42.586356 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:42.594206 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:42.693471 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:42.694353 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:42.697459 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:42.709333 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:42.716500 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:42.727166 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:42.734380 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:42.740792 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:42.747579 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:42.751240 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:42.765973 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:42.768127 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:42.771970 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:42.775574 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:42.795878 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 14:12:42.798553 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:42.800354 21446 maintenance_manager.cc:419] P 43ce4a4c954f4a5f80954b38933c189d: Scheduling FlushMRSOp(814ba2247a314a4797aa0f9f624aaeb6): perf score=0.033393
I20240501 14:12:42.818871 21336 maintenance_manager.cc:643] P 43ce4a4c954f4a5f80954b38933c189d: FlushMRSOp(814ba2247a314a4797aa0f9f624aaeb6) complete. Timing: real 0.018s user 0.013s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":2935,"dirs.run_cpu_time_us":188,"dirs.run_wall_time_us":1653,"drs_written":1,"lbm_read_time_us":1689,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3,"lbm_write_time_us":549,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":144,"threads_started":2,"wal-append.queue_time_us":345}
I20240501 14:12:42.819988 21446 maintenance_manager.cc:419] P 43ce4a4c954f4a5f80954b38933c189d: Scheduling UndoDeltaBlockGCOp(814ba2247a314a4797aa0f9f624aaeb6): 4581 bytes on disk
I20240501 14:12:42.820372 21336 maintenance_manager.cc:643] P 43ce4a4c954f4a5f80954b38933c189d: UndoDeltaBlockGCOp(814ba2247a314a4797aa0f9f624aaeb6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
W20240501 14:12:42.956391 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:12:42.960747 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:12:42.974094 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:12:42.977118 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 14:12:43.058399 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:43.060151 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:43.060778 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:43.064568 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:43.081693 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:43.088205 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:43.097954 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:43.143579 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:43.148412 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:43.149521 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:43.153110 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:43.162288 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:43.173286 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:43.183809 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:43.213565 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 14:12:43.233403 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:43.235978 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:43.240706 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:43.241248 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:43.245121 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:43.249894 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:43.250280 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:43.320698 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:43.322625 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:43.327562 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:43.331851 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:43.331923 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:43.343916 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:43.343924 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:43.360210 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:43.436368 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:43.438817 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:43.438927 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:43.439114 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:43.439417 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:43.442494 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:43.474290 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:12:43.474787 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:43.486466 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 14:12:43.577706 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:43.578871 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:43.587831 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:43.588796 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:43.604285 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:43.619251 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:43.623019 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:43.623715 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:43.628793 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:43.634027 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:43.640345 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:12:43.652734 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:43.653761 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:43.655580 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:43.673698 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:43.745039 21577 maintenance_manager.cc:419] P 778c78105092428bbcde75ab2cdb6a05: Scheduling FlushMRSOp(f6ca088ce3aa4b97aa54c8ce6d700921): perf score=0.033370
I20240501 14:12:43.758909 21467 maintenance_manager.cc:643] P 778c78105092428bbcde75ab2cdb6a05: FlushMRSOp(f6ca088ce3aa4b97aa54c8ce6d700921) complete. Timing: real 0.014s user 0.012s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":174,"drs_written":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4,"lbm_write_time_us":538,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":166,"threads_started":2,"wal-append.queue_time_us":143}
I20240501 14:12:43.759932 21577 maintenance_manager.cc:419] P 778c78105092428bbcde75ab2cdb6a05: Scheduling UndoDeltaBlockGCOp(f6ca088ce3aa4b97aa54c8ce6d700921): 4581 bytes on disk
I20240501 14:12:43.760358 21467 maintenance_manager.cc:643] P 778c78105092428bbcde75ab2cdb6a05: UndoDeltaBlockGCOp(f6ca088ce3aa4b97aa54c8ce6d700921) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
I20240501 14:12:43.792794 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:43.796816 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:43.797168 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:43.809026 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:43.819226 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:43.821182 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:43.824708 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:43.828392 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:43.830457 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:43.836086 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:43.839357 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:43.851552 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 72 similar messages]
I20240501 14:12:43.863247 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:43.881458 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:43.894606 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:43.939596 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:12:44.040241 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:12:44.107645 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:12:44.112097 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:12:44.180338 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 14:12:44.183440 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:44.188714 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:44.196659 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:44.201290 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:44.207172 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:44.209100 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:44.227365 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:44.230840 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:44.241434 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:44.244179 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:44.250559 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:44.251914 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:44.259213 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:44.261612 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:44.324137 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:44.361729 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:44.373863 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:44.375852 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:44.377229 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:44.387065 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:44.390923 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:44.392300 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:44.419178 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:44.433440 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:44.440863 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:44.443722 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:44.445966 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:44.462214 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:44.464263 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:44.505973 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 14:12:44.539278 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:44.543891 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:44.545758 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:44.547219 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:44.547513 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:44.553004 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:44.575682 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:44.663084 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:44.674266 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:44.679769 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:44.681308 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:44.683207 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:44.683593 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:44.686246 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:44.783661 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:44.785082 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:44.793799 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:44.794117 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:44.797289 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:44.809195 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 14:12:44.813568 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:44.825227 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:12:44.830514 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 14:12:44.832224 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:44.860019 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:44.865569 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:44.868063 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:44.868774 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:44.872689 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:44.877274 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 14:12:44.889446 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:44.910967 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:44.964388 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:44.968039 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:44.971364 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:44.972126 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:44.972751 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:44.975500 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:44.984571 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:45.195530 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:45.200987 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:45.202910 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:45.206467 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:45.208014 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:45.215530 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:45.232708 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:45.242645 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:12:45.313262 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 14:12:45.323302 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:45.323318 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:45.326222 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:45.334858 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:45.345434 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:45.346405 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:45.377774 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:45.396044 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:45.396533 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:45.399775 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:45.410586 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:45.411211 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:45.415792 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:45.416585 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:45.443570 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:12:45.447360 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 14:12:45.522817 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:45.530182 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:45.530683 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:45.533752 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:45.541941 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:45.556463 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:45.557430 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:45.560375 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:45.563326 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:45.571208 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:12:45.571965 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 14:12:45.575198 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:45.578220 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:45.590996 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:45.598935 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:45.612813 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 14:12:45.613883 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:45.695652 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:45.698350 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:45.699498 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:45.703976 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:45.709234 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:45.711294 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:45.716984 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:45.809316 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:45.810410 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:45.814941 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:45.817862 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:45.819625 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:45.821813 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:45.836215 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:45.839124 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:45.915216 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:45.919998 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:45.920421 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:45.923235 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:45.932861 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:45.933223 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:45.935376 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:45.997803 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 14:12:46.030525 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:46.031213 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:46.031828 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:46.034183 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:46.035467 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:46.036593 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:46.049328 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:46.076311 21184 maintenance_manager.cc:419] P 5bf81f3d986e428190d3f7727f36d96c: Scheduling FlushMRSOp(11f64b59f83e458f8cf51fd5bcd36fee): perf score=0.033393
I20240501 14:12:46.091121 21074 maintenance_manager.cc:643] P 5bf81f3d986e428190d3f7727f36d96c: FlushMRSOp(11f64b59f83e458f8cf51fd5bcd36fee) complete. Timing: real 0.015s user 0.009s sys 0.005s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":168,"dirs.run_cpu_time_us":285,"dirs.run_wall_time_us":1870,"drs_written":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4,"lbm_write_time_us":473,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":131,"threads_started":2,"wal-append.queue_time_us":138}
I20240501 14:12:46.092063 21184 maintenance_manager.cc:419] P 5bf81f3d986e428190d3f7727f36d96c: Scheduling UndoDeltaBlockGCOp(11f64b59f83e458f8cf51fd5bcd36fee): 4834 bytes on disk
I20240501 14:12:46.092643 21074 maintenance_manager.cc:643] P 5bf81f3d986e428190d3f7727f36d96c: UndoDeltaBlockGCOp(11f64b59f83e458f8cf51fd5bcd36fee) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240501 14:12:46.093418 21184 maintenance_manager.cc:419] P 5bf81f3d986e428190d3f7727f36d96c: Scheduling FlushMRSOp(814ba2247a314a4797aa0f9f624aaeb6): perf score=0.033357
I20240501 14:12:46.108287 21074 maintenance_manager.cc:643] P 5bf81f3d986e428190d3f7727f36d96c: FlushMRSOp(814ba2247a314a4797aa0f9f624aaeb6) complete. Timing: real 0.015s user 0.013s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":43,"dirs.run_cpu_time_us":165,"dirs.run_wall_time_us":1326,"drs_written":1,"lbm_read_time_us":42,"lbm_reads_lt_1ms":4,"lbm_write_time_us":532,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":101,"threads_started":1,"wal-append.queue_time_us":234}
I20240501 14:12:46.109200 21184 maintenance_manager.cc:419] P 5bf81f3d986e428190d3f7727f36d96c: Scheduling UndoDeltaBlockGCOp(814ba2247a314a4797aa0f9f624aaeb6): 4581 bytes on disk
I20240501 14:12:46.109714 21074 maintenance_manager.cc:643] P 5bf81f3d986e428190d3f7727f36d96c: UndoDeltaBlockGCOp(814ba2247a314a4797aa0f9f624aaeb6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
I20240501 14:12:46.149153 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:46.150977 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:46.152966 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:46.161537 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:46.164480 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:46.170650 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:46.171669 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:46.277529 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:46.277801 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:46.278041 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:46.282222 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:46.287199 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:46.294147 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:46.297662 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:46.321164 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:46.361312 21184 maintenance_manager.cc:419] P 5bf81f3d986e428190d3f7727f36d96c: Scheduling FlushMRSOp(55830424a98348e382759c54afdeabcd): perf score=0.033353
I20240501 14:12:46.393239 21074 maintenance_manager.cc:643] P 5bf81f3d986e428190d3f7727f36d96c: FlushMRSOp(55830424a98348e382759c54afdeabcd) complete. Timing: real 0.032s user 0.018s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":42,"dirs.run_cpu_time_us":177,"dirs.run_wall_time_us":2071,"drs_written":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4,"lbm_write_time_us":614,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":100,"threads_started":1,"wal-append.queue_time_us":7636}
I20240501 14:12:46.394656 21184 maintenance_manager.cc:419] P 5bf81f3d986e428190d3f7727f36d96c: Scheduling UndoDeltaBlockGCOp(55830424a98348e382759c54afdeabcd): 4581 bytes on disk
I20240501 14:12:46.397197 21074 maintenance_manager.cc:643] P 5bf81f3d986e428190d3f7727f36d96c: UndoDeltaBlockGCOp(55830424a98348e382759c54afdeabcd) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":360,"lbm_reads_lt_1ms":4}
I20240501 14:12:46.418535 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:46.421356 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:46.423595 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:46.436796 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:46.451835 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:46.452144 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:46.452891 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:46.465778 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:46.476553 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:46.480605 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:46.486310 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:46.492591 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:46.499928 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:46.508852 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:46.576982 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 14:12:46.594313 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:46.595922 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:46.607401 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:46.610375 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:46.610795 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:46.619787 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:46.645854 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:46.656579 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:46.659747 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:46.680809 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:46.683120 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:46.686394 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:46.705957 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 14:12:46.708909 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:46.708976 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:46.755733 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 14:12:46.770673 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:46.772370 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:46.783272 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:46.784757 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:46.788306 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:46.791436 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:46.808156 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:46.827700 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:12:46.917831 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:12:46.973420 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 14:12:47.000555 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:47.001346 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:47.007197 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:47.016196 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:47.017398 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:47.031035 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:47.031337 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:47.122310 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:47.125506 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:47.129783 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:47.131522 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:47.131958 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:47.132894 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:47.142203 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:47.169925 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:47.171355 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:47.174880 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:47.183816 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:47.197060 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:12:47.198316 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 14:12:47.199124 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 14:12:47.199484 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:47.199717 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:47.280467 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:12:47.289193 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:12:47.304987 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:47.308465 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:47.309986 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:47.312589 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:47.321215 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:47.323134 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:47.334980 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:47.345412 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:47.425753 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:47.426824 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:47.428946 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:47.438843 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:47.440205 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:47.456100 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:47.456384 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:47.532230 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 14:12:47.540376 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:47.542964 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:47.553640 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:47.555096 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:47.561698 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:47.562271 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:47.570680 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:47.612000 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:47.612056 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:47.616067 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:47.630008 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:47.637794 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:47.640309 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:47.649746 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:47.741493 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:47.747538 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:47.755928 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 14:12:47.758283 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:47.768265 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:47.770790 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:47.782935 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:47.783365 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:47.783696 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:47.793663 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:47.811224 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:47.812137 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:47.812606 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:47.826640 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:47.835271 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 14:12:47.836192 21446 maintenance_manager.cc:419] P 43ce4a4c954f4a5f80954b38933c189d: Scheduling FlushMRSOp(d63afd081986473aa2715d074b123a93): perf score=0.033338
I20240501 14:12:47.842587 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:47.843489 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:47.860163 21336 maintenance_manager.cc:643] P 43ce4a4c954f4a5f80954b38933c189d: FlushMRSOp(d63afd081986473aa2715d074b123a93) complete. Timing: real 0.023s user 0.014s sys 0.002s Metrics: {"bytes_written":6144,"cfile_init":1,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1326,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":770,"threads_started":2,"wal-append.queue_time_us":174}
I20240501 14:12:47.860981 21446 maintenance_manager.cc:419] P 43ce4a4c954f4a5f80954b38933c189d: Scheduling UndoDeltaBlockGCOp(d63afd081986473aa2715d074b123a93): 4921 bytes on disk
I20240501 14:12:47.861375 21336 maintenance_manager.cc:643] P 43ce4a4c954f4a5f80954b38933c189d: UndoDeltaBlockGCOp(d63afd081986473aa2715d074b123a93) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240501 14:12:47.990231 19816 maintenance_manager.cc:419] P da3c4e948ff747678f18456c134d7ed4: Scheduling FlushMRSOp(db0f01ee72b24ec9b84410cf94ef9759): perf score=0.033349
I20240501 14:12:48.002313 19706 maintenance_manager.cc:643] P da3c4e948ff747678f18456c134d7ed4: FlushMRSOp(db0f01ee72b24ec9b84410cf94ef9759) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":176,"dirs.run_cpu_time_us":206,"dirs.run_wall_time_us":1408,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":502,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":146,"threads_started":2,"wal-append.queue_time_us":164}
I20240501 14:12:48.002985 19816 maintenance_manager.cc:419] P da3c4e948ff747678f18456c134d7ed4: Scheduling UndoDeltaBlockGCOp(db0f01ee72b24ec9b84410cf94ef9759): 4581 bytes on disk
I20240501 14:12:48.003353 19706 maintenance_manager.cc:643] P da3c4e948ff747678f18456c134d7ed4: UndoDeltaBlockGCOp(db0f01ee72b24ec9b84410cf94ef9759) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
W20240501 14:12:48.078289 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:12:48.083796 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:12:48.124191 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:48.125875 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:48.126471 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:48.141027 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:48.141834 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:48.148211 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:48.165023 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:48.203725 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:48.205570 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:48.208541 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:48.213829 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:48.223739 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 14:12:48.224386 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:48.235072 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:48.235297 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:12:48.239557 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 14:12:48.251943 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:48.293689 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:48.295554 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:48.296427 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:48.307458 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:48.310979 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:48.316462 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:48.330152 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:48.388720 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:48.389577 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:48.394491 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:48.399566 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:48.403497 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:48.414355 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:48.424842 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:48.511204 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:48.511623 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:48.513176 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:48.514143 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:48.516379 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:48.519515 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:48.521131 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:48.524701 21577 maintenance_manager.cc:419] P 778c78105092428bbcde75ab2cdb6a05: Scheduling FlushMRSOp(1d408b075d784953a57554bcdf19158a): perf score=0.033383
I20240501 14:12:48.567143 21467 maintenance_manager.cc:643] P 778c78105092428bbcde75ab2cdb6a05: FlushMRSOp(1d408b075d784953a57554bcdf19158a) complete. Timing: real 0.042s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":5864,"dirs.run_cpu_time_us":266,"dirs.run_wall_time_us":1601,"drs_written":1,"lbm_read_time_us":9449,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3,"lbm_write_time_us":595,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":176,"threads_started":2,"wal-append.queue_time_us":152}
I20240501 14:12:48.568070 21577 maintenance_manager.cc:419] P 778c78105092428bbcde75ab2cdb6a05: Scheduling UndoDeltaBlockGCOp(1d408b075d784953a57554bcdf19158a): 4921 bytes on disk
I20240501 14:12:48.568467 21467 maintenance_manager.cc:643] P 778c78105092428bbcde75ab2cdb6a05: UndoDeltaBlockGCOp(1d408b075d784953a57554bcdf19158a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
W20240501 14:12:48.621819 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:48.650091 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:48.655519 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:48.661859 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:48.663276 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:48.670348 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:48.675650 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:48.680265 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:48.691205 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:48.697906 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:48.698839 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:48.702935 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:48.714047 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:48.735050 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:48.735747 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:48.778575 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:12:48.814301 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:48.820796 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:48.822590 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:48.833523 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:48.841020 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:48.843222 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 14:12:48.844961 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:48.881814 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:48.882280 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:48.893296 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:48.896662 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:48.897178 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:48.899484 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:48.906455 21184 maintenance_manager.cc:419] P 5bf81f3d986e428190d3f7727f36d96c: Scheduling FlushMRSOp(cf62635e9a994259b9e293e5eb425b55): perf score=0.033335
I20240501 14:12:48.915927 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:48.929592 21074 maintenance_manager.cc:643] P 5bf81f3d986e428190d3f7727f36d96c: FlushMRSOp(cf62635e9a994259b9e293e5eb425b55) complete. Timing: real 0.023s user 0.014s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"drs_written":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4,"lbm_write_time_us":540,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":154,"threads_started":2,"wal-append.queue_time_us":167}
I20240501 14:12:48.930545 21184 maintenance_manager.cc:419] P 5bf81f3d986e428190d3f7727f36d96c: Scheduling UndoDeltaBlockGCOp(cf62635e9a994259b9e293e5eb425b55): 4834 bytes on disk
I20240501 14:12:48.931023 21074 maintenance_manager.cc:643] P 5bf81f3d986e428190d3f7727f36d96c: UndoDeltaBlockGCOp(cf62635e9a994259b9e293e5eb425b55) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20240501 14:12:48.935225 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:48.946887 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 14:12:49.049333 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:12:49.067028 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:49.192101 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:49.199224 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:49.200942 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:49.203248 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:49.210829 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:49.221606 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:49.222590 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:49.281683 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:49.282523 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:49.282596 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:49.288803 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:49.296816 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:12:49.309917 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 14:12:49.315613 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:49.319458 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:49.328158 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:12:49.375257 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 14:12:49.381199 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:49.383416 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:49.389436 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:49.392571 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:49.401221 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:49.406191 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:49.421813 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 14:12:49.423652 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:49.453672 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:49.472178 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:49.472940 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 14:12:49.473460 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:49.478260 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:49.480105 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:49.484414 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:49.492004 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:49.497007 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:49.533636 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 14:12:49.555425 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:49.559835 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:49.560665 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:49.567945 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:49.569610 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:49.582973 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:49.602542 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:49.685911 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 14:12:49.688395 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:49.690557 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:49.695175 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:49.696205 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:49.698354 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:49.699651 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:49.726956 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:49.819214 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:49.820753 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:49.821231 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:49.828644 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:49.835876 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:49.837446 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:49.845247 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:49.864773 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:49.868088 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:49.888769 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:49.890565 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 14:12:49.896329 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:12:49.897641 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:49.897761 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:49.903982 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:49.908167 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:49.951383 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 14:12:50.007441 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:50.007424 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:50.008500 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:50.009925 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:50.009922 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:50.011140 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:50.018034 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:50.193986 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:50.194739 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:50.201146 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:50.211233 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:50.215756 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:50.216967 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:50.219468 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:50.305416 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:50.325713 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:50.327200 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:50.328138 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:50.328336 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:50.352394 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:50.353979 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:50.361868 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:50.368252 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:50.370468 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:50.383597 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:50.394021 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:50.396868 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:50.414835 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:50.418534 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:50.502297 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 14:12:50.517117 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:50.524693 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:50.527890 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:50.528024 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:50.528596 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:50.535096 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:50.546463 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:50.546795 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:50.551621 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:50.555218 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:12:50.562681 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 14:12:50.564038 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:50.566634 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:50.575086 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 32 similar messages]
I20240501 14:12:50.575237 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:50.600117 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:50.645027 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 14:12:50.683566 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:50.684836 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:50.685052 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:50.689538 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:50.693955 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:50.700317 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:50.701843 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:50.718502 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:50.788734 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:50.792275 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:50.796099 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:50.799263 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:50.807580 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:50.818089 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:50.818655 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:50.898358 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:50.901374 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:50.903860 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:50.904762 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:50.908641 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:50.919271 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:50.919600 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:50.979585 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 14:12:51.010898 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:51.016098 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:51.019078 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:51.019783 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:51.023298 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:51.032305 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:51.046286 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:51.048797 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 14:12:51.121371 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:51.121763 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:51.126123 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:51.128315 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:51.130086 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:51.139405 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:51.140125 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:51.232434 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:51.232535 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:51.237309 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:51.237768 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:51.245891 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:51.261469 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:51.261770 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:51.335590 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:51.361537 21379 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:51.361814 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:51.362183 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:51.363389 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:51.366763 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:51.368434 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:51.379791 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:51.381268 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:51.386030 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:51.388170 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:51.392092 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:51.421202 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:51.430621 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:51.439270 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:51.543136 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:51.548027 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:51.548141 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:51.549582 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:51.550510 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:51.561575 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:51.566891 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:51.585752 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:51.601737 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:51.603646 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:51.605294 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 14:12:51.605562 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:51.605844 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:51.619385 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:51.619385 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:51.712929 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:51.714247 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:51.714903 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:51.721015 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:51.721827 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:51.722530 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:51.740883 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:51.815172 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:12:51.827625 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:51.900350 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:51.901283 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:51.901754 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:51.904821 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:51.905540 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:51.912375 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:51.923604 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:12:51.954754 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:12:51.955289 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 14:12:52.017839 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:52.018630 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:52.025315 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:52.030553 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:52.036432 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:52.037117 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:52.056830 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:12:52.062105 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 14:12:52.092684 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:52.095587 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:52.095757 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:52.098876 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:52.102860 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 14:12:52.110457 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:52.111524 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:52.115675 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:12:52.119472 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:12:52.149338 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:12:52.207944 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:52.212316 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:52.215277 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:52.221800 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:52.231810 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:52.238660 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:52.239768 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:52.243237 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:52.335459 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:52.336405 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:52.339844 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:52.343427 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:52.346385 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:52.371070 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:52.373026 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:52.383354 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 22 similar messages]
W20240501 14:12:52.394486 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 14:12:52.413828 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:52.416483 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:52.420117 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:52.421226 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:52.423687 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:52.424327 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:52.426276 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:52.530221 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:52.530871 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:52.531410 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:52.537284 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:52.538789 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:52.551280 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:52.552970 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:52.562160 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:52.578380 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:52.587136 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:52.588829 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:52.590893 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:52.605736 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:52.622630 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:52.710546 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:52.713930 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:52.715763 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:52.717711 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:52.718438 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:52.734163 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:52.736776 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:52.778481 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:12:52.835366 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 14:12:52.915634 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:52.918378 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:52.923548 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:52.931511 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:52.934062 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:52.934667 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:52.944451 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:53.045822 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:53.052413 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:53.053056 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:53.061625 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:53.085665 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:53.085809 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:53.093530 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:53.097770 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:53.108461 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:53.108829 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:53.114295 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:53.127878 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:53.130092 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:53.139436 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:12:53.168534 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:12:53.175578 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:12:53.214246 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 14:12:53.227056 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:53.227677 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:53.227656 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:53.232867 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:53.236116 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:53.245486 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:53.261487 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:53.285593 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:12:53.338675 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:53.338959 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:53.338986 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:53.339676 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:53.350905 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:53.351722 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:53.353806 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:53.371241 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:53.464342 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:53.466732 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:53.467337 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:53.469882 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:53.482973 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:53.487149 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:53.491326 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:53.513231 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:53.517804 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:53.526384 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:12:53.531414 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 14:12:53.532279 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:53.535276 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:53.538122 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:53.543655 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:53.634784 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:53.636427 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:53.637089 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:53.637486 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:53.641194 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:53.648905 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:53.658427 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:53.711608 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:53.714030 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:53.716313 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:53.723353 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:53.723873 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:53.729537 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:53.733197 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:53.746457 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:53.958209 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:53.964453 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:53.967288 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:53.968209 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:53.970137 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:53.970219 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 14:12:53.970412 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:53.981228 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:53.991752 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 14:12:54.057677 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:12:54.087272 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:54.093396 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:54.095597 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:54.095834 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:54.096100 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:54.099135 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:54.103092 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:54.110826 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:54.121866 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:54.140839 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:54.143916 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:54.150419 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:54.151432 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:54.172560 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:54.199520 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 14:12:54.209175 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:12:54.244210 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:54.251766 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:54.252250 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:54.252669 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:54.252959 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:54.253544 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:54.266039 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:54.273829 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:54.353493 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:54.358165 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:54.358630 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:54.359122 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:54.361526 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:54.372068 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:54.378837 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:54.424453 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 14:12:54.484169 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:54.486171 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:54.491919 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:54.495141 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 14:12:54.502782 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:54.505815 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:54.511276 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:54.511524 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:54.521356 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:54.531837 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:54.532481 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:54.533486 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:54.549971 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:54.553402 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:54.578529 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:54.582783 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:12:54.666797 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 14:12:54.668761 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:54.676887 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:54.682457 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:54.682854 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:54.688658 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:54.696350 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:54.708398 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:12:54.716902 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 14:12:54.717567 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:54.729209 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:54.729976 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:54.737190 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:54.751441 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:54.752547 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:54.765201 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:54.768544 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:54.792670 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 14:12:55.024770 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:55.027885 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:55.033205 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:55.034106 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:55.040030 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:55.053740 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:55.055491 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:55.065485 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 14:12:55.111359 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:55.115581 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:55.117280 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:55.121845 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:55.127642 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:55.135645 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:55.136322 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:55.197113 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:55.198324 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:55.198475 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:55.203359 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:55.209375 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:55.215849 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:55.219121 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:55.227557 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:55.288520 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:55.291103 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:55.295682 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:55.298617 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:55.299494 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:55.309456 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:55.322057 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:55.403694 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:55.405292 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:55.406090 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:55.406777 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:55.410419 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:55.425293 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:55.426383 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:55.481930 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:12:55.516855 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 14:12:55.542049 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:55.543730 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:55.548018 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:55.560122 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:55.560218 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:55.565017 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:55.570317 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:55.578346 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:55.589891 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:55.591779 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:55.592551 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:55.608871 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:55.619283 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:55.642583 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:55.654536 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 14:12:55.712824 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:55.715574 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:55.720925 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:55.724691 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:55.727106 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:55.727424 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 14:12:55.727950 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:55.729522 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:55.747166 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:55.803318 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:55.806257 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 14:12:55.806344 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:55.820849 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:55.821107 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:55.821602 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:55.821710 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:55.834051 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:55.860461 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 14:12:56.083961 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:56.084507 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:56.085942 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:56.090209 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:56.091115 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:56.099571 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:56.108848 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:56.193722 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:56.197682 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:56.198503 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:56.205026 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:56.215253 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:56.217839 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:56.245052 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:56.262718 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:56.266856 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:56.268397 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:56.271212 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:56.271893 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:56.289678 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:56.293498 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:56.312996 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:56.372592 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:56.373414 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:56.375995 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:56.379927 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:56.382282 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:56.393052 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:56.408277 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:56.487996 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:56.488354 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:56.490108 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:56.497989 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:56.498380 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:56.499122 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:56.507597 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:56.609580 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:56.611905 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:56.620131 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:56.621088 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:56.624303 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:56.625963 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:12:56.631451 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 14:12:56.637419 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:56.677306 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 14:12:56.692699 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:56.692910 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:56.693395 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:56.693599 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:56.706710 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:56.710513 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:56.720947 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:56.810451 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:56.814422 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:56.815908 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:56.816176 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:56.828569 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:56.828564 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:12:56.828760 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:56.831570 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:56.838831 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:56.853426 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:56.856527 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:56.856995 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:56.859397 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:56.879135 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 14:12:56.901803 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:56.904217 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:56.919667 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:12:57.011708 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:12:57.201987 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 14:12:57.223520 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:57.231863 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:57.232370 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:57.232929 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:57.233207 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:57.234853 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:57.235514 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:57.237022 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:57.240871 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:57.242630 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:57.267669 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:57.284328 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:57.286512 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:57.289069 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:12:57.392407 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 14:12:57.398243 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:57.406728 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:57.408587 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:57.415237 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:57.417060 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:57.424001 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:57.425352 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:57.431243 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:57.435431 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:57.438061 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:57.439675 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:57.440721 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:57.441983 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:12:57.464383 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 14:12:57.466270 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:57.469101 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:57.526762 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 14:12:57.561427 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:57.563458 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:57.565763 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:57.570360 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:57.585215 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:57.588037 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:57.592522 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:57.666684 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 14:12:57.680522 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:57.681409 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:57.683387 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:57.683601 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:57.691262 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:57.710325 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:57.712993 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:57.808523 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:57.810860 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:57.824009 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:57.825506 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:57.828298 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:57.837311 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:57.842314 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:57.845021 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:57.848210 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:57.852332 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:57.865198 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 14:12:57.868614 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:57.870370 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:57.880210 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:57.899119 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:57.912956 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:58.000782 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:58.001819 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:58.005079 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:58.009456 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 14:12:58.012382 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:58.012429 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:58.015597 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:58.019518 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:58.185931 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:58.190765 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:58.191586 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:58.191812 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:58.203464 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:58.204548 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:58.210973 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:58.260063 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 14:12:58.311254 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:58.313633 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:58.313853 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:58.326973 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:58.329761 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:58.335225 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:58.336594 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:58.378703 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:58.379585 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:58.381397 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:58.382038 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:58.389372 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:58.412479 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:12:58.414942 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:58.415030 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:12:58.441648 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:12:58.442868 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:12:58.467936 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 14:12:58.487917 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:58.495443 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:58.497406 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:58.498448 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:58.499701 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:58.500645 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:58.511947 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:58.571568 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:58.572758 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:58.575932 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:58.583815 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:58.593942 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:58.599884 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:58.606447 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:58.660630 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:58.660964 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:58.664418 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:58.674342 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:58.677103 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:58.677776 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:58.682865 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:58.782923 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:58.790453 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:58.790704 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:58.791725 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:58.791850 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:58.792894 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:58.796411 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:58.903189 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:58.905813 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:58.907094 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:58.912230 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:58.915621 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:58.917793 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:58.918465 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:58.926611 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:12:58.932587 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:12:58.968196 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 14:12:59.024916 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:59.025282 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:59.026322 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:59.028982 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:59.032255 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:59.039507 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:59.055181 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:59.128011 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:59.129348 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:59.129482 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:59.137161 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:59.137976 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:59.139216 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:59.162155 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:12:59.164698 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240501 14:12:59.243026 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:59.244180 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:59.245354 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:59.245787 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:59.248191 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:59.251914 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:59.276319 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:59.365598 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:59.366942 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:59.376502 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:59.379958 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:59.380131 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:59.380882 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:59.383627 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:59.385972 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:59.395326 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:59.410737 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:12:59.410842 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 14:12:59.425729 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:59.431936 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:59.441340 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:59.443799 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:59.453675 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:59.460731 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:12:59.522436 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:12:59.544255 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:59.544801 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:59.552831 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:59.553318 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:59.568938 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:59.577076 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:59.579247 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 14:12:59.580785 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:59.584757 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:59.594673 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 14:12:59.595981 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:59.597302 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:59.598646 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:59.617630 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:59.627719 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:59.629204 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:12:59.689432 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:12:59.722704 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:59.724637 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:59.724884 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:59.738610 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:59.743309 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:12:59.750259 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:59.754729 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:12:59.824935 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 14:12:59.931851 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:59.934053 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:59.936722 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:59.940776 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:12:59.941548 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:12:59.946097 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:12:59.954356 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:00.011368 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 14:13:00.043188 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:00.050017 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:00.050421 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:00.059168 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:00.062425 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:00.065999 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:00.090904 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:00.107906 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:00.108964 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:00.112473 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:00.115710 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:00.124096 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:00.130060 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:00.145470 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:00.147706 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:00.231500 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:00.232149 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:00.234061 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:00.235698 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 14:13:00.238718 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:00.239071 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:00.250088 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:00.263912 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:00.338717 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:00.343717 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:00.344295 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:00.355301 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:00.355559 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:00.363471 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:00.366430 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:00.416827 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 14:13:00.461441 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:00.465273 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:00.465348 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:00.468334 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:00.486600 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:00.488150 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:00.490154 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:00.522151 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:00.526937 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:00.531193 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:00.540841 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 14:13:00.547760 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:00.558909 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:00.563114 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:00.567487 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:00.625911 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:00.631311 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:00.632134 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:00.632177 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:00.639816 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:00.640527 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:00.649250 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:00.666213 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:00.714140 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:00.716245 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:00.718567 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:00.727283 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:00.728500 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:00.731720 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:00.740005 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:00.873911 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 14:13:00.963025 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:00.969005 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:00.970017 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:00.971633 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:00.979884 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:00.987010 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:01.004803 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:01.036314 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 14:13:01.097287 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:01.098326 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:01.098585 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:01.098780 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:01.102934 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 14:13:01.103214 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:01.105998 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:01.106148 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 14:13:01.108997 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:01.117883 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:01.124845 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:01.125006 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:01.125768 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:01.133965 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:01.135088 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:01.153373 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:01.161604 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:01.255860 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:01.258495 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:01.259343 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:01.259709 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:01.260620 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:01.261001 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:01.284703 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 14:13:01.285892 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:01.363934 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:01.363934 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:01.368008 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:01.368723 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:01.369850 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:01.371755 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:01.392319 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:01.495519 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:01.496311 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:01.497978 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:01.510178 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:01.511129 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:01.517930 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:01.518354 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:01.525146 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:01.525666 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:01.539681 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:01.555275 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:01.557374 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:01.563488 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:01.572564 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 14:13:01.586009 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:01.613286 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:01.682653 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:01.684453 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:01.688475 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:01.693207 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:01.696092 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:01.697422 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:01.704903 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:01.712157 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:01.718938 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:01.721957 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 14:13:01.727989 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:01.730306 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:01.733757 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:01.746290 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:01.761377 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:01.954337 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 14:13:02.062449 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:02.065991 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 14:13:02.066848 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:02.070549 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:02.077560 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:02.086809 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:02.096657 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:02.111578 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:02.112702 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:02.113107 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:02.118633 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:02.125839 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 14:13:02.135922 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:02.136801 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:02.140583 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:02.161813 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:02.202442 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:02.248621 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:02.255357 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:02.263427 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:02.264710 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:02.265898 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:02.265967 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:02.266144 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:02.267921 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:02.283568 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:02.305883 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:02.312994 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 14:13:02.313103 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:02.325580 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:02.327718 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:02.334048 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:02.408290 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:02.409808 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:02.416337 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:02.418279 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:02.426251 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:02.438433 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:02.439335 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:02.526614 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:02.528035 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:02.531334 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:02.531965 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:02.538393 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:02.558423 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:02.559427 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:02.586361 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 14:13:02.587211 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:13:02.634547 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 14:13:02.659421 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:02.662951 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:02.667476 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:02.672959 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:02.680286 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:02.683954 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:02.685413 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:02.686622 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:02.690272 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:02.704703 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:02.705222 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:02.715008 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:02.727831 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:02.730119 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:02.734032 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:02.812619 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 14:13:02.846679 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:02.847290 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:02.853773 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:02.859627 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:02.868218 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:02.868961 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:02.881417 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:03.044945 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:03.045765 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:03.046581 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:03.049108 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:03.052656 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:03.063992 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:03.080268 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:03.107540 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 14:13:03.181815 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:03.188935 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:03.195034 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:03.203562 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:03.209033 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:03.209329 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:03.213196 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:03.216050 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:03.232887 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:03.241712 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:03.244271 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:03.256932 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:03.257151 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:03.275761 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:03.278050 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:03.291478 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 14:13:03.358244 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:03.363572 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:03.372853 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:03.388115 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:03.388537 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:03.399911 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:03.417191 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 14:13:03.417492 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:03.427098 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:03.438871 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:03.441409 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:03.444366 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:03.445906 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:03.450795 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:03.470336 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:03.532338 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:03.532564 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:03.534190 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:03.535883 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:03.537163 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:03.541312 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:03.552830 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:03.607264 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 14:13:03.639606 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:03.640017 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:03.644073 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:03.647445 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:03.651002 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:03.653880 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:03.681010 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:03.694586 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:13:03.695772 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 14:13:03.757251 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:03.759903 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:03.761761 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:03.765054 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:03.773638 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:03.780251 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:03.787777 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:03.805791 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:13:03.856057 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 14:13:03.876277 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:03.879168 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:03.881137 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:03.885608 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:03.886469 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:03.894196 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:03.915997 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:03.990434 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:03.994230 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:03.999207 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:04.004269 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:04.012252 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:04.013808 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:04.013865 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:04.100646 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 14:13:04.109702 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:04.113570 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:04.120604 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:04.120709 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:04.121949 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:04.140615 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:04.147130 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:04.232513 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:04.236649 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:04.245535 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:04.248205 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 72 similar messages]
I20240501 14:13:04.251789 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:04.251802 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:04.260082 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:04.263173 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:04.266989 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:04.275347 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:04.286712 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:04.291317 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:04.306124 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:04.307072 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:04.312080 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:04.325407 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 14:13:04.330440 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:04.403321 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:04.409066 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:04.415582 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:04.417136 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:04.418395 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 14:13:04.420758 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:04.422386 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:04.430403 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:04.491199 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:04.491932 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:04.493394 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:04.495136 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:04.501502 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:04.512516 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:04.521065 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:13:04.534049 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:04.611526 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:04.611934 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:04.615487 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:04.621588 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:04.622401 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:04.622872 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:04.641901 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:04.729121 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:04.729219 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:04.730530 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:04.738284 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:04.738554 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:04.758581 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:04.759898 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:04.822778 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 14:13:04.842823 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:04.845126 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:04.845984 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:04.857725 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:04.862279 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:04.864797 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:04.879025 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:04.883211 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:13:04.916442 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 14:13:04.917397 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:04.961848 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:04.964831 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:04.966918 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:04.967809 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:04.971637 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:04.987969 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 14:13:04.993515 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:05.002082 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:05.088569 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:05.095222 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:05.095945 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:05.097723 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:05.098518 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:05.104247 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:05.121063 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 14:13:05.121285 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:05.201210 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:05.201902 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:05.212970 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:05.214941 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:05.231935 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:05.248948 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:05.255882 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:05.260553 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:05.269420 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:05.274986 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:05.275862 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:05.282933 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:05.297808 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:05.301731 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:05.337842 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:13:05.378374 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:05.409005 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:05.414203 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:05.421010 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:05.424154 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:05.428515 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:05.441830 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:05.444479 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:05.454527 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:05.458189 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:05.460696 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:05.462707 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:05.463189 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:05.491119 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:05.498513 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:05.581331 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:05.584455 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:05.585673 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:05.588089 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:05.589524 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:05.611550 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:05.614290 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:05.792760 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:05.794708 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:05.800616 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:05.802045 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:05.804867 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:05.806059 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:05.820065 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:05.843073 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:13:05.843559 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:13:05.843636 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:13:05.874711 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 14:13:05.925416 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:05.925741 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:05.933058 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:05.937085 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:05.940886 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:05.945641 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:05.950114 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:05.969928 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:05.972668 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:05.973599 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:05.981359 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:05.990478 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:05.993320 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:06.003324 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:06.100355 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:06.104087 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:06.105034 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:06.106122 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:06.107506 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:06.117249 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:06.121073 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:06.203218 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:06.209584 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 14:13:06.212528 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:06.213469 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 14:13:06.221486 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:06.226486 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:06.227946 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:06.230070 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:06.239302 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 14:13:06.244936 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:06.246066 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:06.329165 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:06.338521 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:06.339381 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:06.343032 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:06.346001 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:06.347491 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:06.372890 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:06.398391 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:06.410701 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 14:13:06.414361 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:06.418820 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:06.419131 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:06.420117 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:06.435101 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:06.444983 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:06.486900 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 14:13:06.503545 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:06.504292 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:06.508147 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:06.508790 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:06.512697 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:06.526329 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:06.539659 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:06.577806 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:06.583680 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:06.585901 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:06.589807 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:06.598446 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:06.600373 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:06.623574 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:06.641353 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:06.830462 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:06.830679 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:06.839476 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:06.840202 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:06.840262 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:06.850102 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:06.866640 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:06.959097 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:06.966203 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:06.966897 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:06.971607 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:06.977052 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:06.978565 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:06.984150 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:06.998428 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:07.001361 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 14:13:07.008970 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:07.009675 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:07.015648 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:07.040469 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:07.040788 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:07.060277 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:07.071657 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 14:13:07.130040 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:07.131347 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:07.131894 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:07.141759 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:07.147702 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:07.150743 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:07.165827 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:07.198511 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:13:07.204629 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 14:13:07.241569 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:07.246152 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:07.246729 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:07.247345 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:07.249763 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:07.262182 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:07.264775 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:07.342242 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 14:13:07.359093 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:07.359563 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:07.360577 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:07.361699 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:07.371244 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:07.379451 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:07.387449 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:07.436311 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:07.439383 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:07.444007 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:07.448732 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:07.456949 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:07.458663 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:07.486860 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:07.515853 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 14:13:07.526326 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:07.529747 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:07.534781 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:07.534819 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:07.550561 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:07.560523 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:07.569422 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:07.592660 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:07.620257 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:07.621500 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:07.627830 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:07.629079 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:07.630527 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:07.630625 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:07.634650 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:07.711103 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 14:13:07.736316 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 14:13:07.858872 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:07.860172 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:07.864877 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:07.873343 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:07.876410 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:07.880615 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 14:13:07.881639 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:07.906936 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:08.003814 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:08.010210 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:08.011535 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:08.014509 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:08.020653 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:08.024423 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:08.034960 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:08.034963 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:08.036175 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:08.041188 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:08.042542 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:08.071635 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:08.074301 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:08.098510 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:08.160975 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:08.181636 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:08.182551 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:08.186452 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:08.187857 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:08.191468 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:08.205199 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 14:13:08.205968 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:08.207113 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:08.239380 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:13:08.248548 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 14:13:08.296818 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:08.300958 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:08.307967 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:08.311067 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:08.315513 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:08.317101 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:08.337309 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:08.412289 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:08.424490 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:08.431759 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:08.435896 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:08.436918 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:08.453016 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:08.458007 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:08.461760 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:08.472340 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:08.475451 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:08.484555 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:08.487468 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:08.493288 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:08.502159 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:08.609223 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:08.609783 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:08.615691 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:08.621608 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:08.637751 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:08.643411 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:08.643791 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:08.647138 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:08.647208 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:08.648370 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:08.650617 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:08.656534 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:08.677433 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:08.689424 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:08.701869 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 14:13:08.708091 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:13:08.764214 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:13:08.781064 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:13:08.874610 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:13:08.878309 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 14:13:08.994038 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:09.002467 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:09.009403 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:09.019493 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:09.019610 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:09.029819 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:09.040578 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:09.050143 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:09.052369 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:09.059386 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:09.065881 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:09.066006 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:09.080857 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:09.081353 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:09.196414 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:09.197400 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:09.212491 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:09.216233 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:09.219004 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:09.222678 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:09.223215 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:09.226437 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:09.235950 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:09.239711 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:09.257306 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:09.274456 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 14:13:09.275171 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:09.282763 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:09.290853 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:09.293054 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:09.370383 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:09.374089 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:09.381958 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:09.382640 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:09.395689 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:09.400815 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:09.402043 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:09.415822 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 14:13:09.496670 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:09.496793 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:09.497014 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:09.501052 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:09.503165 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:09.522964 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:09.523855 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:09.565591 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:13:09.566805 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 14:13:09.614454 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:09.620764 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:09.626250 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:09.629002 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:09.633080 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:09.635532 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:09.650347 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:09.691476 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:09.695123 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:09.700524 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:09.701635 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:09.703874 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:09.705834 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:09.706892 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:09.778610 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:13:09.795820 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:09.800801 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:09.803192 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:09.805922 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:09.806746 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:09.808964 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:09.812413 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 14:13:09.828660 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:09.847106 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:13:09.919206 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 14:13:10.016717 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:10.017746 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:10.025810 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:10.026321 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:10.026528 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:10.027472 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:10.041469 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:10.078263 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 14:13:10.133071 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:10.144691 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:10.144932 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:10.154392 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:10.163612 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:10.165735 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:10.174500 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:10.200338 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:10.204133 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:10.207356 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:10.209225 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:10.222265 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:10.232409 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:10.233000 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:10.251358 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:13:10.321295 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:10.325049 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:10.327358 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:10.336275 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:10.338785 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:10.351030 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:10.351198 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:10.373160 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:10.374292 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:10.376649 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:10.395110 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:10.396337 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:10.412818 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:10.416304 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:10.445379 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:10.526746 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:10.532413 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:10.534655 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:10.536235 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:10.537968 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:10.539175 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:10.539759 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:10.588580 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:13:10.632707 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 14:13:10.640408 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:10.642273 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:10.649645 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:10.652199 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:10.664710 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:10.669909 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:10.673044 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:10.739246 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:10.741815 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:10.744354 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:10.749457 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:10.752427 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:10.761631 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:10.767262 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:10.767450 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 14:13:10.768761 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:10.861454 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:10.864078 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:10.864203 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:10.870174 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:10.870498 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:10.871870 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:10.889062 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:10.919337 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 14:13:10.981401 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:10.983067 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:10.984730 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:11.003024 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:11.003998 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:11.006309 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:11.007023 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:11.042105 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 14:13:11.107488 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:11.108819 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:11.112630 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:11.114399 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:11.115237 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 14:13:11.119520 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:11.124255 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:11.128060 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:11.220180 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 14:13:11.223302 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:11.225220 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:11.225493 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:11.240573 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:11.241864 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:11.242206 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:11.263988 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:11.275316 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:13:11.275977 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:11.304226 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:11.308420 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:11.309110 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:11.309486 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:11.320479 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:11.324534 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:11.341616 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:11.389113 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:11.394022 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:11.404315 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:11.411114 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:11.413544 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:11.426411 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:11.436249 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:11.488287 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:11.488875 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:11.492784 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:11.494684 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:11.507501 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:11.508419 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:11.518453 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:11.521735 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 14:13:11.606695 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:11.607014 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:11.607798 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:11.617326 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:11.619141 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:11.629184 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:11.629187 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:11.728889 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:11.729499 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:11.731480 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:11.742645 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:11.744417 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:11.757830 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:11.767649 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:11.842104 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:11.846509 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:11.847455 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:11.850121 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:11.863365 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:11.865096 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:11.865680 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:11.879616 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:11.955577 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:11.955860 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:11.961033 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:11.961663 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:11.970551 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:11.982554 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 14:13:11.983827 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:11.991657 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:12.065325 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:13:12.065361 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 14:13:12.078820 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:12.079957 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:12.081779 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:12.082233 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:12.087200 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:12.091838 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:12.092531 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:12.101003 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 14:13:12.205758 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:12.208518 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:12.208582 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:12.210729 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:12.211776 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:12.217440 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:12.237135 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:12.240160 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:12.241964 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:12.246232 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 14:13:12.252621 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:12.260892 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:12.263188 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:12.263595 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:12.263689 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:12.355193 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:12.394078 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:12.394174 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:12.400769 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:12.402727 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:12.411916 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:12.420924 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:12.426666 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:12.432614 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:12.433298 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:12.435386 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:12.449643 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:12.455485 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:12.481384 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:12.489311 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:12.572978 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:12.573592 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:12.573915 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:12.576157 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:12.579906 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:12.592021 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:12.608829 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:12.614955 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 14:13:12.779790 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:12.785251 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:12.789346 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:12.792872 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:12.794287 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:12.812152 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:12.823498 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:12.913682 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240501 14:13:12.924674 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:12.933522 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:12.935209 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:12.939345 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:12.946867 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:12.947794 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:12.963121 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:12.965220 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:12.966817 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:12.970304 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:12.974714 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:12.975741 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:12.976361 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:13.000484 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:13.003456 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:13.106073 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:13.108904 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:13.111380 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:13.114492 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:13.116209 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:13.120214 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:13.136765 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:13.136859 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:13:13.207028 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 14:13:13.222435 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:13.231935 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:13.232335 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:13.233350 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:13.234074 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:13.247879 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:13.256237 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:13.339648 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:13.342337 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:13.348510 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:13.355614 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:13.356844 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:13.358304 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:13.377979 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 14:13:13.394942 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:13.405174 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:13.411700 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:13.424765 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:13.428349 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:13.431339 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:13.435426 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:13.435720 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:13.438822 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:13.459880 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:13:13.478798 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 14:13:13.552778 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:13.555496 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:13.555850 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:13.568679 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:13.586859 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:13.588991 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:13.590551 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:13.595400 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:13.596652 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:13.597980 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:13.598793 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:13.612661 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:13.630342 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:13.640908 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 14:13:13.644536 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:13.746320 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:13:13.884730 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20240501 14:13:13.918582 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:13.944133 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:13.946537 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:13.949273 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:13.959949 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:13.960716 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:13.964468 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:13.981391 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:14.009223 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:14.020655 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:14.020663 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:14.021806 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:14.037170 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:14.037580 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:14.041977 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:14.126709 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:14.130643 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:14.137012 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:14.138144 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:14.148273 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:14.148805 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:14.155218 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:14.205340 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:14.208420 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:14.215932 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:14.227705 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:14.236340 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:14.247797 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:14.250885 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:14.293483 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:14.295400 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:14.298352 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:14.302814 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:14.305487 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:14.308881 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:14.319803 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 14:13:14.326253 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:14.382762 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240501 14:13:14.414548 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:14.418704 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:14.418798 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:14.420336 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:14.429131 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:14.438535 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:14.440459 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:14.442154 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:13:14.480340 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:13:14.503520 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 14:13:14.536012 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:14.537660 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:14.546003 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:14.546048 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:14.548959 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:14.564144 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:14.564412 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:14.576942 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 14:13:14.615846 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:14.616010 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:14.620412 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:14.624079 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:14.626461 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:14.645238 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:14.647313 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:14.732283 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:14.733122 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:14.733951 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:14.740893 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:14.741590 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:14.761512 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:14.763767 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:14.772322 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 14:13:14.871995 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:14.872279 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:14.876919 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:14.878185 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:14.884805 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:14.885943 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:14.895382 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:14.902818 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:13:14.962622 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:13:14.970541 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 14:13:14.978495 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 14:13:14.984488 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:14.985333 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:14.988862 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:14.991705 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:14.993449 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:15.000440 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:15.019019 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:15.115454 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:15.118686 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:15.122851 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:15.134840 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:15.146076 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:15.148042 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:15.149089 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:15.149128 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:15.150897 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:15.156078 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:15.158602 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:15.175828 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:15.189110 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:15.196123 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:15.302899 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:15.305711 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:15.305820 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:15.314558 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:15.317494 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:15.321311 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:15.328560 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:15.330276 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:13:15.354620 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:15.361264 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:15.367928 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:15.375316 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:15.379860 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:15.384302 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:15.391340 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:15.412822 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:13:15.480820 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 14:13:15.486615 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:15.487905 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:15.488503 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:15.489859 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:15.494638 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:15.501322 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:15.501946 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:15.651247 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 14:13:15.664278 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:15.666862 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:15.675452 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:15.680945 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:15.693007 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:15.696357 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:15.696581 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:15.730293 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:13:15.785988 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 14:13:15.800061 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:15.800662 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:15.806634 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:15.809619 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:15.820143 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:15.832202 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:15.845122 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:15.848794 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:15.866673 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:15.871902 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:15.879767 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:15.885222 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:15.888041 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:15.896000 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:15.898378 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:15.990322 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:15.990993 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:15.993350 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:15.993408 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:16.004104 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:16.007987 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:16.018196 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:16.036607 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 14:13:16.100875 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:16.103029 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 14:13:16.106611 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:16.107101 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:16.117676 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:16.119971 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:16.122545 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:16.135478 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:16.205669 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:16.215224 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:16.220114 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:16.225769 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:16.232395 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:16.235898 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:16.237182 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:16.291092 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:16.291853 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:16.292346 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:16.300218 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:16.314460 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:16.314942 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:16.319234 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:16.357479 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 14:13:16.380623 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:16.381047 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:16.404163 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:16.404515 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:16.407706 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:16.413983 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:16.431277 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:16.468152 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:13:16.469080 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:16.473789 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:16.474146 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:16.474552 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:16.477443 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:16.479421 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:16.480053 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:16.488091 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:16.663499 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 14:13:16.723935 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:16.729318 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:16.730159 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:16.731534 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:16.735322 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:16.757987 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 14:13:16.760298 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:16.773422 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:16.833969 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:16.836556 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:16.846215 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:16.856503 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:16.861757 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:16.876391 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:16.878509 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:16.910583 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:16.914554 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:16.921687 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:16.922621 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:16.928061 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:16.930042 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:16.943207 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:16.951210 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:17.024163 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:17.024508 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:17.030077 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:17.030140 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:17.031270 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:17.032542 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:17.043905 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:17.132563 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:17.134959 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:17.135437 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:17.135900 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:17.140033 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:17.143898 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 14:13:17.161499 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:17.161563 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:17.219458 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 14:13:17.259136 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:17.259867 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:17.261267 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:17.266346 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:17.270545 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:17.273365 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:17.289140 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:17.332535 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:17.335681 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:17.338218 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:17.344383 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:17.345407 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:17.348632 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:17.352267 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 14:13:17.360891 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:17.413219 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:13:17.443271 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:17.459470 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:17.465018 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:17.466558 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:17.469125 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:17.478284 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:17.479555 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:17.497988 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:17.503540 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:17.513190 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:17.517381 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:17.518738 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:17.527180 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:17.527419 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:17.536828 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:17.643364 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 14:13:17.839614 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:17.843156 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:17.846215 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:17.859230 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:17.859751 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:17.877063 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:17.891145 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:17.897940 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:17.900027 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:17.915719 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:17.920562 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:17.925014 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 14:13:17.932256 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:17.934597 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:17.954039 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:17.955121 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:13:17.965669 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:13:17.988365 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 14:13:18.033767 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:18.034670 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:18.035984 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:18.042122 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:18.042580 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:18.045859 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:18.054282 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:18.109697 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:18.113276 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:18.116609 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:18.117010 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:18.118546 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:18.126564 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:18.135217 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:18.224583 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:18.224977 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:18.231343 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:18.235250 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:18.235865 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:18.238740 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:18.260385 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:18.283617 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 14:13:18.375586 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:18.384160 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:18.385046 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:18.388031 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:18.391525 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:18.394124 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 14:13:18.396517 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:18.404026 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:18.415769 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:18.415817 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:18.422207 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:18.422909 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:18.428386 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 14:13:18.435794 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:18.447885 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:18.449141 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:18.463660 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:18.546029 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:18.549697 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:18.559828 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:18.563356 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:18.563827 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:18.570506 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:18.573184 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 14:13:18.586942 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:18.615865 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:18.618383 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:18.622473 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:18.626371 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:18.626865 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:18.628263 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:18.638861 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:18.842761 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:13:18.885319 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:13:18.886073 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 14:13:18.897974 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:18.899319 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:18.903151 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:18.904515 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:18.911237 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:18.930743 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:18.934217 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:18.975821 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:18.981839 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:18.986117 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:18.986238 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:18.991839 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:19.005592 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:19.009572 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:19.017758 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:19.082449 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:19.083600 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:19.090605 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:19.093853 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:19.094208 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:19.094439 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:19.095970 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:19.124574 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 14:13:19.168848 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:19.169749 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:19.173287 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:19.175854 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:19.184953 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:19.196485 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:19.197548 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:19.252274 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:19.255915 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:19.256966 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:19.257756 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:19.260026 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:19.266796 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:19.283174 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:19.365177 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:19.366771 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:19.368132 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:19.372134 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:19.384488 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:19.392007 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:19.399415 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:19.425798 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:13:19.468003 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 14:13:19.487936 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 14:13:19.491508 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:19.495934 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:19.500808 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:19.507364 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:19.512398 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:19.512748 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:19.540362 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:19.549019 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:19.552165 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:19.566505 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:19.567489 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:19.578193 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:19.578951 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:19.584338 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:19.649941 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 14:13:19.675724 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:19.676362 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:19.678515 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:19.690259 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:19.698421 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:19.707437 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:19.719372 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:19.784965 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 14:13:19.881764 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:19.882017 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:19.886500 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:19.891285 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:19.896522 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:19.902055 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:19.904021 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:19.994961 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:13:20.012071 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:20.013104 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:20.015889 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:20.023734 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:20.025197 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:20.033005 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:13:20.052897 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:20.056421 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:20.057956 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:20.060626 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:20.067368 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:20.075541 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:20.082427 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:20.104521 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:20.108886 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:20.109547 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:20.203675 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:20.210037 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:20.212472 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:20.215538 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:20.218225 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:20.229549 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:20.231797 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:20.249243 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 14:13:20.257081 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:20.257664 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:20.266512 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:20.272406 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:20.272500 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:20.273802 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:20.283758 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:20.373762 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:20.375150 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:20.376621 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:20.379243 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:20.384161 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:20.405301 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:20.406529 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:20.468983 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:13:20.475685 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 14:13:20.491427 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:20.494460 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:20.497206 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:20.498347 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:20.501819 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:20.522111 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:20.522991 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:20.609055 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:20.612370 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:20.612848 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:20.614964 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:20.619083 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:20.625537 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:20.630271 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:20.651096 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:13:20.667204 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:13:20.726483 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:20.729943 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:20.732141 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:20.733413 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:20.738898 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:20.739183 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:20.770423 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:20.842903 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:20.843394 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:20.847623 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:20.858420 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:20.859663 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:20.864409 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:20.869367 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:20.880242 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:20.891449 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:13:20.929853 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 14:13:20.972458 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:20.974171 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:20.978976 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:20.984838 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:20.988831 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:21.003497 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:21.004470 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:21.006863 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 14:13:21.105046 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:21.106467 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:21.112407 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:21.122392 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:21.124926 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:21.125640 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:21.143675 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:21.154963 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:21.161665 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:21.169816 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:21.172307 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:21.176396 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:21.179929 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:21.182001 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:21.199348 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 14:13:21.285902 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:21.289304 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:21.290254 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:21.290439 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:21.309059 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:21.311661 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:21.323154 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:21.332406 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:21.346123 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:21.356241 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:21.359266 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:21.365190 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:21.365674 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:21.369413 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:21.407244 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 14:13:21.475857 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:21.477193 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:21.477253 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:21.477943 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:13:21.478677 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:21.483484 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:21.491405 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:21.492647 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:21.501400 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:21.635236 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 14:13:21.645345 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:21.648919 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:21.651849 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:21.653684 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:21.655441 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:21.667455 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:21.679421 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:21.697201 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 14:13:21.762382 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:21.767357 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:21.774897 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:21.779235 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:21.780468 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:21.789718 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:21.803511 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:21.850217 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:21.851661 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:21.855664 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:21.861564 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:21.864584 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:21.868656 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:21.871768 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:21.938557 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 14:13:21.972272 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:21.972301 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:21.980630 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:21.981148 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:21.983460 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:21.988602 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:21.988693 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:21.994539 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:22.059216 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:13:22.067783 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 14:13:22.094483 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:22.101842 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:22.102244 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:22.102916 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:22.109638 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:22.112766 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:22.137655 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:22.162539 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:22.163187 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:22.167454 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:22.180377 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:22.181491 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:22.184358 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:22.211439 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:22.275981 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:22.278424 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:22.278983 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:22.281698 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:22.287861 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:22.296895 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:22.313191 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:22.351269 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:22.352481 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:22.356909 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:22.357765 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:22.363235 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:22.365116 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:22.365700 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:22.405907 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:22.449785 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:22.450414 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:22.450628 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:22.454367 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:22.456310 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:22.465601 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:22.478406 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:22.502847 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 14:13:22.706881 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:22.712805 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:22.715548 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:22.721001 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:22.721931 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:22.723563 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:22.724849 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:22.784245 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:22.817348 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:22.819633 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:22.825219 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:22.834026 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:22.835783 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:22.836762 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:22.837371 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:22.853930 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 14:13:22.890872 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:22.911885 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:22.923768 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:22.927583 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:22.927874 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:22.929239 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:22.949548 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:23.008380 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 14:13:23.028151 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:23.028247 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:23.036265 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:23.038784 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:23.045547 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:23.046284 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:23.053704 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:23.143213 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:23.143791 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:23.145550 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:23.153986 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:23.154402 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 14:13:23.155941 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:23.180176 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:23.180675 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:23.273841 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:23.278954 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:23.285652 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:23.290242 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:23.299240 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:23.299732 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:23.310662 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:23.328348 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:23.331450 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:23.333309 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:23.337390 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:13:23.341360 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 14:13:23.342633 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:13:23.343955 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 14:13:23.346767 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:23.352042 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:23.355535 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:23.359791 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:23.382290 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 14:13:23.458546 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:23.471386 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:23.476331 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:23.481375 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:23.485934 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:23.486039 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:23.489217 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:23.490067 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:23.502729 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:23.509295 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:23.510354 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:23.511330 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:23.515884 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:23.532853 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:23.731104 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:13:23.781872 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:23.848953 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:23.851720 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:23.852353 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:23.857951 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:23.866097 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:23.879192 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:23.882525 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:23.882934 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:23.903025 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:23.907212 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:23.917163 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:23.922120 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:23.923394 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:23.938156 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:24.044947 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:24.046187 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:24.047010 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:24.049393 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:24.049970 19748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:24.050217 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:24.051393 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:24.068954 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:24.074851 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:24.075192 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:24.092314 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:24.101251 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:24.120211 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:24.128856 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:24.185151 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:13:24.193977 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 14:13:24.205899 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:24.206261 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:24.215296 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:24.216763 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:24.220506 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:24.232641 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:24.248169 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:24.332373 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:24.333232 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:24.335803 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:13:24.337848 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:24.342365 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:24.343659 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:24.345420 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:24.355638 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:24.358189 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 14:13:24.375658 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:24.440279 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:24.440902 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:24.452834 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:24.459750 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 14:13:24.471451 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:24.475878 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:24.484163 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:24.489746 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:24.490061 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:24.492694 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:24.493095 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:24.516801 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:24.519863 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:24.521134 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:24.522843 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:24.612329 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:24.614622 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:24.618988 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:24.626272 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:24.627146 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:24.633719 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:24.641654 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:24.678506 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 14:13:24.697252 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:13:24.769207 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:13:24.776290 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 14:13:24.824851 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:24.824864 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:24.826478 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:24.833132 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:24.835458 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:24.838733 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:24.847402 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:24.850262 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:24.965245 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:24.970335 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:24.973196 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:24.987258 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:24.996304 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:24.999991 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:25.007196 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:25.009517 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:25.014174 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:25.021731 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:25.024763 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:25.029587 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:25.035212 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:25.046665 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:25.154928 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:25.156777 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:25.157398 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 14:13:25.158373 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:25.165758 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:25.166225 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:25.167213 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:25.168211 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:25.175199 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:25.187518 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:25.200609 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:13:25.215953 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:25.225435 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:25.231446 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:25.233896 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:25.235564 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:25.326063 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:25.326684 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:25.329473 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:25.337874 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:25.340052 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:25.347575 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:25.349682 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:25.365700 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:25.428568 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 14:13:25.448659 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:25.451267 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:25.453027 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:25.454200 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:25.456686 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:25.464493 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:25.476356 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:25.556140 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:25.561678 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:25.561753 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:25.562593 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:25.565387 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:25.566397 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:25.598268 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:25.605589 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 14:13:25.680742 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:25.685582 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:25.687193 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:25.691370 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:25.691634 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:25.692173 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:25.693944 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 14:13:25.702636 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:25.756656 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 14:13:25.803882 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:25.805577 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:25.814467 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:25.819746 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:25.819962 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:25.820919 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:25.824566 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:25.870844 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 14:13:25.931015 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:25.931737 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:25.935897 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:25.937601 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:25.947494 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:25.948688 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:25.956286 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:25.960860 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:26.024268 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:13:26.027206 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 14:13:26.062125 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:26.062206 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:26.069307 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:26.069728 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:26.072258 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:26.072841 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:26.073552 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:26.095669 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:26.099908 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:26.102795 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:26.107551 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:26.108587 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:26.133374 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:26.153038 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:26.221947 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:26.227638 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:26.228461 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:26.231122 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:26.246362 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:26.247476 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:26.271183 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:26.298812 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:26.303820 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:26.309352 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:26.311232 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:26.314460 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:26.318423 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:26.324316 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:26.391098 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:26.418331 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:26.422304 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:26.423743 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:26.431489 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:26.439962 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:26.443794 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:26.452883 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:26.568492 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:26.569597 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:26.572062 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:26.579219 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:26.582314 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:26.601548 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:26.603639 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:26.629598 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 14:13:26.686992 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:26.689066 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:26.689564 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:26.699016 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:26.699241 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:26.699793 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 14:13:26.701329 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:26.715212 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:26.721437 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:13:26.789000 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:26.795718 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:26.796836 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:26.797925 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:26.800863 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:26.801949 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:26.801970 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:26.819861 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:26.842492 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 14:13:26.917688 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:26.922226 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:26.924263 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:26.925216 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:26.927546 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:26.932021 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:26.939415 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:27.041996 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:27.047097 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:27.056182 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:27.058535 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:27.062618 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:27.065824 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:27.081346 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:27.091442 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 14:13:27.092435 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:27.097707 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:27.100178 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:27.100243 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:27.107244 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:27.111356 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 14:13:27.124176 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:27.128296 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:27.223112 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 14:13:27.231323 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:27.239815 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:27.247872 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:27.248584 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:27.257084 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:27.262883 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:27.273006 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:27.273902 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:27.275527 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:27.277451 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:27.280514 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:27.282655 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:27.294740 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:27.305565 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:27.313520 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:27.337514 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 14:13:27.403687 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:27.407505 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:27.408772 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:27.413050 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:27.414044 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:27.435452 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:27.444702 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:27.608736 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:27.609558 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:27.610424 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:27.614913 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:27.623204 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:27.632155 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:27.634609 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:27.724802 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:27.735142 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:27.745154 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:27.745501 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:27.745905 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:27.750155 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:27.750384 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:27.769789 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:27.772154 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 14:13:27.809619 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:27.813006 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:27.815176 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:27.818421 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:27.819548 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:27.829396 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:27.831398 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:27.908812 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 14:13:27.931685 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:27.933547 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:27.934448 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:27.936244 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:27.937970 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:27.957628 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:27.967170 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:28.022980 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 14:13:28.049094 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:28.050529 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:28.061028 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:28.062438 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:28.063228 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:28.063449 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:28.073129 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:28.084666 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 14:13:28.134390 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:28.134538 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:28.138666 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:28.141088 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:28.147374 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:28.155675 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:28.155870 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:28.162456 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:13:28.176293 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:28.220934 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:28.222854 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:28.225514 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:28.230788 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:28.241369 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:28.243434 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:28.252583 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:28.256455 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 14:13:28.304745 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:28.310343 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:28.316560 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:28.317092 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:28.318486 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:28.325217 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:28.339200 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:28.390348 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:28.390635 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:28.394534 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:28.408223 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:28.410725 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:28.412340 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:28.424767 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:28.629575 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:28.633127 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:28.634721 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:28.641016 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:28.643877 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:28.645054 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:28.647533 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:28.688566 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 14:13:28.695740 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 14:13:28.761706 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:28.763098 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:28.768400 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:28.772764 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:28.773542 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:28.788586 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:28.801302 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:28.807740 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:28.825520 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:28.827421 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:28.834187 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:28.834957 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:28.838775 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:28.853350 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:28.854581 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:28.936780 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:28.941226 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:28.945205 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:28.945561 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:28.949054 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:28.951438 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:28.956393 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:29.049424 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:29.052090 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:29.052886 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:29.056517 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:29.068001 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:29.068730 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:29.076678 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:29.093672 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 14:13:29.172571 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:29.183064 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:29.183887 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:29.186465 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:29.186726 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:29.187247 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:29.188292 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:29.230170 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:29.231318 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:29.240216 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:29.242602 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:29.244596 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:29.255649 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:29.259518 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:29.319586 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:13:29.328474 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 14:13:29.342202 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:29.342856 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:29.345692 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:29.347402 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:29.348074 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:29.357067 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:29.379765 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:29.413738 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:29.414016 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:29.418928 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:29.425789 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:29.434159 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:29.446888 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:29.447593 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:29.459704 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:13:29.485232 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:13:29.634335 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 14:13:29.672161 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:29.678632 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:29.679796 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:29.680083 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:29.687927 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 72 similar messages]
I20240501 14:13:29.689914 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:29.705857 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:29.711894 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:29.734925 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:13:29.747699 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 14:13:29.795221 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:29.797010 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:29.798207 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:29.804256 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:29.806392 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:29.815697 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:29.822324 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:13:29.838519 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 14:13:29.842288 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:29.888068 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:29.890966 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:29.894395 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:29.896890 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:29.897735 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:29.899911 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:29.919626 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:30.014461 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:30.015949 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:30.016875 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:30.018483 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:30.027734 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:30.029148 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:30.037099 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:30.125103 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:30.127797 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:30.132666 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:30.132745 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:30.134032 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:30.142032 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:30.147617 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:30.195919 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 14:13:30.241611 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:30.242759 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:30.245767 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:30.251724 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:30.256866 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:30.257265 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:30.271510 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:30.294054 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:30.304335 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:30.307917 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:30.313562 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:30.313650 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:30.318030 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 14:13:30.319854 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:30.320659 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:30.322259 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:30.359580 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:13:30.426584 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:30.429468 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:30.431202 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:30.440052 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:30.442386 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:30.463898 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:30.464608 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:30.469668 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:30.474262 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:30.475391 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:30.488668 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:30.502533 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:30.513556 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:30.528643 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:30.537279 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:13:30.576124 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:13:30.585587 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:13:30.725922 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:30.820222 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:30.821949 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:30.829567 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:30.831542 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:30.835752 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:30.840226 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:30.841755 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:30.843773 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:30.861141 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:30.869077 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:30.875849 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:30.895754 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:30.897637 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 14:13:30.902751 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:30.907861 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:30.984959 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:30.985885 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:30.988981 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:30.991238 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:31.003095 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:31.009537 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:31.036736 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:31.054567 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:31.061774 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:31.064477 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:31.065105 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:31.074476 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:31.079166 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:31.094792 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:31.157411 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 14:13:31.158041 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 14:13:31.172730 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:31.172977 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:31.174244 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:31.176106 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:31.176151 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:31.176654 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:31.205948 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:31.235257 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:31.286620 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:31.295245 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:31.297155 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:31.299608 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:31.306845 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:31.317658 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:31.318217 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:31.374445 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:31.385922 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:31.389712 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:31.396262 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:31.396517 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:31.401109 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:31.416359 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:31.463589 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:31.465466 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:31.476052 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:31.479341 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:31.487027 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:31.489488 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:31.499361 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:31.550361 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:31.555322 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:31.559432 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:31.559587 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:31.568368 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:31.577826 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:31.580090 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:31.684039 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:13:31.724191 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:13:31.730245 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 14:13:31.794979 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:31.795303 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:31.798918 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:31.802999 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:31.802996 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 14:13:31.803393 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:31.805440 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 14:13:31.807193 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:31.824723 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:31.920104 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:31.925172 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:31.938818 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:31.941484 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:31.942159 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:31.943641 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 14:13:31.947177 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:31.950538 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:31.951239 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:31.956609 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:31.958774 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:31.964329 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:31.980862 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:31.988255 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:31.990198 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:32.105355 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 14:13:32.106604 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:32.110783 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:32.112210 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:32.118345 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:32.120541 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:32.121270 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:32.147248 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:32.147745 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:32.155299 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:32.159036 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:32.174801 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:32.184471 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:32.185107 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:32.185528 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:32.196935 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:32.280184 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:32.286648 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:32.287703 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:32.290014 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:32.298542 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:32.303570 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:32.303659 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:32.396435 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:32.397601 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:32.402551 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:32.412225 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 14:13:32.412384 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:32.412400 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:32.415172 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:32.429512 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:32.512930 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:32.513914 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:32.514454 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:32.517668 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:32.529196 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:32.530543 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:32.539933 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 14:13:32.551893 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:32.633670 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:32.634721 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:32.637586 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:32.641726 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:32.661253 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:32.663846 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:32.665848 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:32.705010 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:32.745368 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:32.749060 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:32.750617 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:32.755297 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:32.768026 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:32.775481 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:32.789688 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 14:13:32.792330 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:32.854395 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 14:13:32.881505 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:32.883143 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 14:13:32.884317 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:32.889207 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:32.892040 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:32.899714 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:32.908412 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:32.916280 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:32.928442 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 14:13:33.025622 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:33.029933 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:33.038619 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:33.043893 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:33.050004 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:33.060932 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:33.062547 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:33.065034 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:33.067121 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:33.081660 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:33.082717 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:33.083422 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:33.102079 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:33.115166 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:33.166350 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:33.208693 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:33.212384 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:33.221097 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:33.227969 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:33.230245 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:33.240579 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:33.248895 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:33.268002 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:33.270354 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:33.275310 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:33.275456 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:33.278983 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:33.281401 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:33.302174 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:33.354827 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 14:13:33.382350 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:33.391974 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:33.392030 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:33.392220 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:33.398540 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:33.399523 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:33.411422 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:33.425077 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:13:33.560544 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:33.579684 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:33.580435 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:33.585753 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:33.587590 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:33.597309 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:33.612673 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:33.613201 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:33.653601 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 14:13:33.715588 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:33.724571 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:33.727324 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:33.742591 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:33.755214 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:33.755694 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:33.764294 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:33.768282 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:33.774425 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:33.783279 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:33.786022 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:33.795625 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:33.812579 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:33.822858 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 14:13:33.834408 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:33.911821 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:33.913947 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:33.922133 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:33.923540 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:33.926129 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:33.937549 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:33.948226 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:33.948338 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:34.031994 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:34.032490 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:34.033468 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:34.037567 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:34.040634 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:34.047483 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:34.047621 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 14:13:34.057916 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:34.137501 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:34.143618 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:34.148249 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:34.154966 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:34.159452 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:34.169934 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:34.171494 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:34.210227 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:34.213044 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:34.213645 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:34.214195 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:34.215768 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:34.216014 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:34.234344 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:34.257913 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 14:13:34.333770 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:34.338912 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:34.339934 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:34.343274 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:34.346673 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:34.353523 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:34.366250 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:34.366657 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:34.375361 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:34.382985 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:34.383731 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:34.387362 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:34.413641 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:34.417037 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:34.513299 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:13:34.523484 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:13:34.566632 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:13:34.682595 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 14:13:34.716872 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:34.718459 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:34.718930 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 68 similar messages]
I20240501 14:13:34.719756 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:34.722949 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:34.725432 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:34.734324 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:34.742636 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:34.750890 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:34.761780 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:34.762481 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:34.768456 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:34.772630 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:34.776108 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:34.795224 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:34.901820 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:34.905488 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:34.905977 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:34.913012 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:34.921535 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:34.930711 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:34.931265 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:34.936928 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:34.969677 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:34.976897 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:34.977412 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:34.977972 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:34.977975 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:34.978619 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:35.008769 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:13:35.013211 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:13:35.078806 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:35.081305 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:35.085387 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:35.085963 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:35.086935 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:35.087651 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:35.088433 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:35.111529 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:13:35.145823 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 14:13:35.193866 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:35.202986 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:35.203413 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:35.203491 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:35.205600 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 14:13:35.206702 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:35.218739 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:35.228983 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:35.292158 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:35.292184 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:35.294981 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:35.301234 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:35.308049 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:35.321099 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:35.321341 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:35.377244 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:35.377230 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:35.383349 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:35.395958 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:35.400610 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:35.403025 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:35.406345 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:35.466660 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:35.467478 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:35.469903 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:35.475481 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:35.479264 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:35.479503 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:35.485693 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:35.503775 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:13:35.568153 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:13:35.586668 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:13:35.628288 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:13:35.703280 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 14:13:35.714794 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:35.719036 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:35.719934 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:35.721228 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:35.721344 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:35.728281 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:13:35.730897 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 14:13:35.731618 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:35.740353 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:35.847106 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:35.850826 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:35.857553 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:35.860663 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:35.869320 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:35.883543 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:35.886821 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:35.889331 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:35.903877 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:35.917904 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:35.918810 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:35.922799 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:35.929509 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:35.938192 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:35.969174 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:13:36.010489 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:36.031953 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:36.042425 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:36.046033 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:36.047572 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:36.061269 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:36.062417 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:36.077581 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:36.082036 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:36.091583 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:36.093138 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:36.093683 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:36.095628 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:36.110451 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:36.114886 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:36.205543 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:36.207572 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:36.208109 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:36.213217 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:36.213953 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:36.233731 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:36.248838 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:36.322333 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:36.323092 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:36.330142 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:36.330147 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:36.332161 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:36.332965 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:36.342147 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:36.394999 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:13:36.425716 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:13:36.432857 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:36.438493 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:36.443881 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:36.446894 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:36.450716 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:36.459355 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:36.459865 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:36.539862 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:36.540387 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:36.545956 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:36.546257 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:36.551384 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:36.554556 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:36.555944 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:36.587260 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:13:36.618813 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 14:13:36.657133 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:36.659080 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:36.665357 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:36.665730 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:36.672227 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:36.679853 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:36.699766 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:36.782318 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:36.784523 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:36.786688 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:36.796029 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:36.800621 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:36.808099 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:36.818245 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:36.893503 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:36.898901 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:36.903666 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:36.904773 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:36.907820 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:36.909909 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:36.916772 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:36.973151 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:36.973142 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:36.974068 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:36.977819 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:36.984786 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:37.000058 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:37.004108 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:37.062567 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:37.064772 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 14:13:37.064908 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:37.070139 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:37.082984 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:37.084560 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:37.089979 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:37.090775 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:37.111984 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:13:37.112782 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:37.162472 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:37.167366 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:37.167881 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:37.171443 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:37.178722 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:37.189235 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 14:13:37.189841 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:37.190135 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:37.279248 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:37.280570 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:37.287020 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:37.289877 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:37.293673 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:37.294245 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:37.294873 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:37.338902 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 14:13:37.384792 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:37.386920 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:37.391804 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:37.391822 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:37.393463 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:37.394173 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:37.431474 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:37.504905 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:37.509238 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:37.510169 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:37.512521 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:37.513947 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:37.530838 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:37.532305 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:37.549453 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:13:37.588582 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 14:13:37.619314 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:37.620682 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:37.622624 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:37.622723 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:37.626565 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:37.627859 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:37.647976 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:37.710862 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:13:37.718341 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 14:13:37.738081 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:37.745121 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:37.745992 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:37.747246 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:37.751838 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:37.764293 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:37.764456 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:37.866565 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 14:13:37.868412 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:37.883941 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:37.884827 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:37.885716 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:37.886322 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:37.897253 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:37.899897 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:37.903699 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:37.917975 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:37.934022 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:37.935935 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:37.951161 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:37.953290 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:37.971925 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:38.044317 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 14:13:38.048470 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:38.049417 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 14:13:38.054718 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:38.054745 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:38.057684 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:38.061120 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:38.077348 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:38.087620 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:38.108335 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:13:38.111831 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 14:13:38.123036 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:38.125524 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:38.128031 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:38.132355 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:38.132997 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:38.153239 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:38.163635 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:38.170001 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 14:13:38.217530 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:38.218075 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:38.218261 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:38.222055 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:38.226929 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:38.228941 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:38.230109 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:38.240553 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 14:13:38.450649 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:38.451651 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:38.456502 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:38.464499 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:38.466634 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:38.469715 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:38.471813 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:38.495133 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:38.573779 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:38.576851 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:38.577003 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:38.578521 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:38.578784 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:38.582626 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:38.605345 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:38.608618 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:38.616307 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:38.624128 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:38.628901 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:38.638999 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:38.639688 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:38.640123 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:38.746845 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:38.746940 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:38.747251 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:38.747407 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:38.749861 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:38.762235 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:38.768455 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:38.794323 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 14:13:38.853469 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:38.853729 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:38.854319 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:38.855149 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:38.858336 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:38.867759 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:38.904461 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:38.975487 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:38.975984 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:38.976433 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:38.978612 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:38.986562 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:39.001752 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:39.001905 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:39.026206 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:39.054862 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:39.055552 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:39.056495 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:39.065436 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:39.068959 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:39.077344 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:39.082603 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:39.095988 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 14:13:39.148382 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:39.148644 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:39.151857 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:39.156675 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:39.163219 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:39.167968 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:39.172111 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:39.186683 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 14:13:39.219136 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:39.220456 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:39.231518 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:39.235121 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:39.238469 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:39.241192 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:39.264873 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:39.476655 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:39.477082 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:39.481333 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:39.485766 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:39.487854 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:39.501690 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:39.516244 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:39.539310 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:39.600399 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:39.610505 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:39.611626 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:39.628145 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:39.628417 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:39.628803 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:39.640705 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:39.664286 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:39.665409 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:39.672807 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:39.675171 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:39.687744 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:39.689899 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:39.694695 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:39.757825 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240501 14:13:39.758512 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:13:39.767416 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 14:13:39.804689 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:39.806295 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:39.807699 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:39.808166 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:39.814837 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:39.829505 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:39.842033 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:39.917564 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:39.920539 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:39.921244 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:39.930886 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:39.933195 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:39.946287 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:39.955268 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 14:13:39.959522 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:40.022939 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 14:13:40.049799 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:40.049978 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:40.052402 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:40.054550 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:40.069761 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:40.070052 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:40.086736 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:40.095027 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:40.098076 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:40.105118 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:40.110494 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:40.111011 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:40.112946 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:40.120869 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:40.125625 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:40.177393 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 14:13:40.216583 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:40.221602 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:40.225112 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:40.227512 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:40.230118 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:40.231153 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:40.251935 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 14:13:40.265908 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:40.286219 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:40.287329 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:40.290671 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:40.295123 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:40.300885 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:40.306944 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:40.307090 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:40.416035 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:13:40.422616 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:13:40.473412 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:13:40.534358 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 14:13:40.558552 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:40.560865 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:40.570559 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:40.573875 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 14:13:40.578629 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:40.588644 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 14:13:40.601990 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:40.601986 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:40.602468 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:40.648274 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:40.651269 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:40.656380 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:40.663919 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:40.680276 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:40.683416 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:40.683862 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:40.699388 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:40.736863 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:40.739101 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:40.739858 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:40.745749 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:40.749224 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:40.757706 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:40.763914 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:40.774520 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 14:13:40.844733 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:40.845506 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:40.845779 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:40.846330 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:40.851184 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:40.874305 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:40.875821 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:40.952303 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:40.954981 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:40.955677 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:40.956306 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:40.958195 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:40.970057 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:40.986645 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:41.013033 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:13:41.065565 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:41.067152 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:41.069137 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:41.075759 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:41.078081 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:41.087314 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:41.090392 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:41.135115 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:41.140380 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:41.142731 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:41.147189 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:41.149509 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:41.152675 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:41.162966 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:41.182052 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:41.224779 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:41.227595 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:41.233021 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:41.235234 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:41.241756 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:41.245244 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:41.256125 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:41.312687 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:41.313088 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:41.317421 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:41.323220 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:41.329007 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:41.330047 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:41.339987 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:41.446527 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 14:13:41.550503 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:41.554654 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:41.556840 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:41.561091 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:41.563982 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:41.572121 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:41.572899 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:41.599269 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:13:41.608808 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 14:13:41.681753 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:41.688383 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:41.690539 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:41.695138 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:41.704393 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:41.709491 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:41.715193 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:41.729876 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:41.755396 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:41.755883 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:41.767359 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:41.771270 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:41.780761 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:13:41.785287 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:41.788743 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:41.873420 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:41.877368 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:41.879662 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:41.880060 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:41.885696 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:41.894556 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:41.899236 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:41.982038 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:41.983810 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:41.984198 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:41.984450 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:41.994047 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:42.016505 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:42.016909 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:42.063477 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:42.108229 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:42.113365 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:42.113637 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:42.115296 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:42.125403 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:42.128103 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:42.140184 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 14:13:42.141166 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:42.141484 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:42.150497 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:42.157128 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:42.168851 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:42.176939 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:42.185444 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:42.185907 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:42.199549 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:13:42.240306 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 14:13:42.289276 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:42.292474 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:42.296604 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:42.303125 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:42.313632 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:42.316288 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:42.316437 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:42.317255 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:42.320847 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:42.324088 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:42.329324 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:42.334180 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:42.338606 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:42.351747 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:42.571547 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:13:42.625669 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:13:42.629030 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 14:13:42.673600 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:42.674010 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:42.676817 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:42.677752 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:42.688082 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:42.701809 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:42.708200 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:42.709980 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:42.712085 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:42.719609 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 14:13:42.719718 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:42.720827 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:42.723134 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:42.727612 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:42.742704 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:42.846705 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:42.848673 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:42.863289 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:42.869732 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:42.873597 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:42.883359 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:42.905151 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:42.905616 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:42.922534 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 14:13:42.923527 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:42.924342 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:42.933841 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:42.939260 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:42.955916 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:42.956166 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:42.994388 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 14:13:43.042770 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:43.043644 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:43.045526 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:43.049538 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:43.060257 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:43.060997 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 14:13:43.062867 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:43.074071 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:43.108284 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:13:43.119424 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 14:13:43.169027 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:43.169303 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:43.172590 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:43.178951 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:43.183609 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:43.196184 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:43.212522 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:43.213814 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:13:43.240048 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 14:13:43.280515 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:43.281319 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:43.287259 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:43.287685 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:43.296363 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:43.309007 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:43.312002 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:43.313261 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 14:13:43.345407 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:43.354811 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:43.356527 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:43.358700 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:43.379230 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:43.384527 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:43.385039 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:43.438941 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:43.443588 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:43.447479 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:43.447762 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:43.449218 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:43.450448 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:43.470391 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:43.706214 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:43.713001 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:43.713670 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:43.716331 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:43.723915 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:43.737838 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:43.742430 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:43.757472 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:43.812655 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:43.814464 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:43.820237 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:43.822450 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:43.828827 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:43.829108 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 14:13:43.835230 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:43.853021 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:43.910936 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:43.915453 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:43.915825 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:43.916079 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:43.918619 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:43.942683 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:43.945366 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:44.006666 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:44.008009 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:44.014555 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:44.014640 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:44.016884 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:44.032064 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:44.035987 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:44.088191 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:44.090198 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:44.090378 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:44.090849 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:44.094002 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:44.094278 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:44.101361 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:44.199584 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:44.199981 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:44.200436 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:44.207295 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:44.207713 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:44.212920 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:44.243156 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:44.268432 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:13:44.280927 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:44.321316 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:44.325354 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:44.334282 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:44.335426 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:44.355346 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:44.361428 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:44.368746 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:44.382305 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:44.388366 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:44.390195 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:44.394464 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:44.399072 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:44.408540 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:44.416473 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:44.442839 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240501 14:13:44.504835 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:44.506424 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:44.507272 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:44.510699 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:44.521507 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:44.531069 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:44.531988 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:44.698092 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:44.700910 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:44.703146 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:44.711580 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 14:13:44.716140 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:44.719153 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:44.720268 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:44.723577 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:44.783339 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 14:13:44.784981 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:44.837579 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:44.840162 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:44.845445 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:44.850751 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:44.852810 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:44.867980 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:44.873566 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:44.874009 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:44.887583 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:44.895771 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:44.898449 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:44.916158 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:44.916822 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:44.917711 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:44.957252 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:13:44.978190 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 14:13:45.010346 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:45.015645 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:45.023576 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:45.026768 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:45.033946 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:45.040169 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 14:13:45.048722 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:45.063202 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:45.063248 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:45.067986 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:45.077826 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:45.080281 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:45.084918 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:45.088790 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:45.093441 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:45.184193 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:45.189630 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:45.190601 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:45.190608 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:45.199383 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:45.204389 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:45.209153 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:45.230033 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:13:45.230293 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 14:13:45.298525 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:45.301481 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:45.304925 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:45.314311 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:45.316478 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:45.323410 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:45.329109 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:45.345758 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:13:45.381011 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:13:45.382622 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 14:13:45.415457 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:45.418517 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:45.428128 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:45.428371 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:45.430670 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:45.442456 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:45.462337 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:45.538133 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:45.539255 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:45.539268 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:45.543002 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:45.547094 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:45.549593 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:45.560113 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:45.652834 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:45.653018 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:45.658941 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:45.664095 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:45.667033 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:45.669245 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 14:13:45.669798 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:45.682688 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:45.693279 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:13:45.760433 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 14:13:45.777179 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:45.785804 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:45.788183 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:45.788564 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:45.798797 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:45.799285 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:45.822485 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 14:13:45.823503 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:45.829159 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:45.908776 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:45.914237 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:45.914762 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:45.915166 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:45.929073 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:45.951792 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:45.958567 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:45.958756 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 14:13:45.969903 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:45.970860 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:45.973632 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:45.978577 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:45.989915 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:46.004637 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:46.005393 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:46.098111 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:46.104583 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:46.109344 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:46.111454 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:46.129051 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:46.145720 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:46.151226 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:46.151320 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:46.162139 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:46.164358 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:46.170570 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:46.182857 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:46.190733 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:13:46.191774 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:46.194656 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:46.284062 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:46.286939 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:46.287781 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:46.288467 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:46.289556 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:46.292348 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:46.292798 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:46.315367 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:46.345368 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 14:13:46.471380 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:46.472229 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:46.474694 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:46.478721 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:46.483577 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:46.491909 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:46.506561 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:46.586418 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:46.595510 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:46.596354 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:46.606943 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:46.608747 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:46.620538 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:46.621174 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:46.659549 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:46.660728 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:46.665827 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:46.668136 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:46.668380 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:46.668898 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:46.673305 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:46.779764 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:46.780076 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:46.781672 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:46.788321 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:46.790745 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:46.799876 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:46.816534 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:46.883584 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:46.900810 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:46.901082 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:46.915280 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:46.915428 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:46.927235 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:46.928293 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:46.928750 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:46.995386 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:47.010972 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:47.011312 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:47.011924 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:47.017230 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:47.023515 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:47.031989 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 14:13:47.034967 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:47.090875 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:13:47.094005 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:47.097383 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:47.098933 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:47.102190 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:47.105479 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:47.112113 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:47.120344 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:47.175974 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:47.179643 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:47.179821 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:47.179965 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:47.180188 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:47.184335 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:47.191612 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:47.258713 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 14:13:47.267880 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:47.272617 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:47.274506 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:47.287289 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:47.287686 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:47.293779 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:47.295850 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240501 14:13:47.311228 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:47.316920 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:13:47.355962 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 14:13:47.497536 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:47.500560 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:47.502796 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:47.503975 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 14:13:47.504498 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:47.509778 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:47.521952 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:47.522315 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:47.581324 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 14:13:47.626569 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:47.633426 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:47.633697 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:47.638480 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:47.639865 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:47.647270 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:47.657502 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:47.677975 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:47.682757 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:47.686625 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:47.687032 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:47.699915 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 14:13:47.702795 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:47.703315 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:47.707705 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:47.772998 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:47.812285 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:47.819316 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:47.819916 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:47.822363 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:47.824398 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:47.831809 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:47.835965 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:47.836516 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:13:47.876147 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:13:47.883755 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 14:13:47.940518 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:47.943852 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:47.951356 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:47.954428 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:47.960914 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:47.961921 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:47.970813 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:47.987908 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:13:48.027700 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 14:13:48.041934 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:48.042389 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:48.042618 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:48.049427 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:48.064333 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:48.075668 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:48.079319 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:48.114743 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:48.120831 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:48.131680 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:48.132465 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:48.137246 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:48.141494 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:48.160933 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:48.200368 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:48.205952 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:48.207334 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:48.207813 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:48.210249 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:48.219949 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:48.231436 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:48.301314 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:48.302603 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:48.302984 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:48.308866 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:48.311196 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:48.314082 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:48.321504 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:13:48.324602 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 14:13:48.332054 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:48.386466 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:48.535140 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:48.537644 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:48.538220 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:48.541792 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:48.551535 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:48.562124 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:48.565446 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:48.652788 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:48.655404 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:48.664296 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:48.667286 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:48.668237 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:48.680914 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:48.706821 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:48.715207 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:48.719211 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:48.720329 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:48.723590 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:48.724670 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:48.727438 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:48.745708 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:48.809198 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 14:13:48.828347 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:48.831833 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:48.833010 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:48.835202 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:48.835317 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:48.835930 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:48.837244 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:48.919121 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:48.939152 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:48.945917 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:48.946049 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:48.946974 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:48.947367 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:48.947777 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:48.964222 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:49.070950 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:49.080600 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:49.081454 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:49.081724 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:49.088989 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:49.095451 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:49.097002 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:49.100404 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:49.100487 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:49.120262 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:49.121698 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:49.123081 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:49.136085 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:49.145537 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:49.254997 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:49.257073 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:49.264245 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:49.265151 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:49.266368 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:49.269091 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:49.272414 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:49.280017 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:49.288287 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:49.308020 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:49.310305 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:49.318821 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:49.333993 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:49.335000 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:49.469456 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:49.616439 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:49.620442 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:49.624034 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:49.624797 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:49.632840 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:49.656195 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:49.657491 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 14:13:49.672394 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:49.674469 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:49.681579 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:49.688206 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:49.699792 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:49.701996 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:49.702399 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:49.709296 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:49.809144 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 14:13:49.817656 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:49.825780 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:49.826112 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:49.831254 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:49.834209 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:49.838595 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:49.843293 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:49.872905 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:49.875770 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 14:13:49.876483 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 14:13:49.892235 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:49.898685 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:49.899034 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:49.902825 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:49.903239 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:49.903568 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:49.919027 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 14:13:50.006850 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:50.007313 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:50.009943 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:50.010098 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:50.032819 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:50.036624 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:50.040511 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:50.092726 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:50.123797 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:50.127125 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:50.127936 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:50.131516 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 14:13:50.136652 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:50.139544 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:50.145483 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:50.148561 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:50.234603 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:50.235198 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:50.237967 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:50.239156 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:50.240185 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:50.247330 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 14:13:50.260865 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:50.276619 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:50.300841 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:50.310783 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:50.311018 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:50.313988 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:50.314888 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:50.320760 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 14:13:50.321763 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:50.342460 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:50.358942 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:13:50.358951 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:13:50.359923 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 14:13:50.393327 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:50.396467 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:50.397486 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:50.402465 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:50.404453 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:50.407131 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:50.414244 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:50.500569 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:13:50.514034 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:13:50.600231 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 14:13:50.624260 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:50.624569 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:50.635223 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:50.635267 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:50.636003 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 14:13:50.647306 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:50.652244 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:50.655233 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:50.671590 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 14:13:50.736140 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:50.742493 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:50.744081 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:50.755780 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:50.756319 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:50.759972 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:50.776964 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:50.807289 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:50.817123 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:50.819718 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:50.820605 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:50.823132 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:50.826763 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:50.836494 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:50.914433 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:50.917395 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:50.920764 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:50.934670 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:50.934998 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:50.952613 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:50.965999 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:50.971480 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:50.980249 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:50.983537 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:50.990782 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:50.994664 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:51.013826 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:51.015517 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:51.030565 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:51.112251 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:51.112965 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:51.116314 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:51.126015 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:51.127429 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:51.128360 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:51.143182 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:51.177778 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:13:51.228682 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:51.229739 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:51.231684 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:51.233474 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:51.236927 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:51.255962 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:51.255962 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:51.315424 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 14:13:51.324148 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:51.324353 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:51.326879 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:51.329713 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:51.335021 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:51.336650 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:51.354775 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:51.409142 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:51.435972 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:51.436355 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:51.439281 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:51.443872 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:51.453490 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:51.459242 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:51.476399 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:51.555212 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:51.556394 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:51.558434 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:51.560783 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:51.561340 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:51.577991 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:51.596230 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:51.673454 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:51.684403 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:51.684514 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:51.685887 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:51.690245 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:13:51.701815 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:51.709081 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:51.710355 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:51.796610 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:51.798547 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:51.800341 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:51.810176 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:51.827785 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:51.830149 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:51.845183 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:51.849586 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:51.862299 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:51.868767 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:51.871757 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:51.881254 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:51.884418 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:51.885109 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:51.900609 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:51.983897 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:51.985661 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:51.996749 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:52.001777 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:52.003391 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:52.014016 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:52.037451 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:52.042073 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:52.045290 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:52.054672 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:52.064096 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:52.067940 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:52.070472 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:52.071272 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:52.088315 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 14:13:52.161712 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:52.167591 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:52.170081 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:52.170634 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:52.177318 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:52.187952 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:52.190418 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:52.304010 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:13:52.304741 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:13:52.332410 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:13:52.346807 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:52.347113 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:52.351208 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:52.351781 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:52.355521 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:52.361259 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:52.374004 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:52.384196 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:13:52.424893 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 14:13:52.464313 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:52.465701 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:52.470746 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:52.471397 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:52.484999 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:52.491328 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:52.503010 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:52.537392 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:52.543404 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:52.543668 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:52.543910 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:52.546296 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:52.546351 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:52.547648 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:52.655303 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:52.658296 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:52.659123 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:52.661286 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:52.663188 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:52.666218 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:52.691906 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:52.700392 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:13:52.703998 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:13:52.736394 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 14:13:52.776027 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:52.781315 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:52.794740 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:52.795639 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:52.796725 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:52.798875 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:52.835429 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:52.837469 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 14:13:52.847169 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:52.851239 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:52.859814 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:52.860014 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:52.861649 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:13:52.874748 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 14:13:52.875507 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:52.879299 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:52.900954 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:52.901185 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:13:52.914853 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:13:52.944602 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 14:13:52.982892 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:52.992319 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:52.993070 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:52.995483 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:52.996035 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:53.013286 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:53.016906 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:53.032161 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:53.038947 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:53.051242 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:53.067204 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:53.079222 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:53.081730 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:53.095279 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:53.178916 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:53.182747 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:53.183477 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:53.185387 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:53.185676 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:53.187359 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:53.192628 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:53.279592 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:13:53.286986 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:13:53.360669 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:53.397095 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:53.398373 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:53.398758 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:53.404532 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:53.404882 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:53.406152 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:53.426198 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:53.521819 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:53.523785 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:53.530879 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:53.535666 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:53.538069 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:53.538666 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:53.563661 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:53.589249 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:53.592116 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:53.600654 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:53.607620 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:53.620224 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:53.621369 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:53.622709 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:53.708087 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:53.709136 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:53.717296 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:53.717553 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:53.718503 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:53.718703 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:53.723423 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:53.793681 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:53.821502 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:53.823434 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:53.826964 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:53.832183 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:53.834815 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:53.835335 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:53.864123 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:53.934233 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:53.936131 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:53.937114 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:53.937919 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:53.942142 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:53.942142 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:53.967332 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:54.007193 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:54.007404 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:54.011463 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:54.030619 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:54.031543 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:54.035993 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:54.037261 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:54.046639 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 14:13:54.095621 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:54.100589 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:54.101028 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:54.101711 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:54.116309 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:54.120191 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:54.141322 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:54.188995 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:54.188995 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:54.195235 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:54.195899 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:54.211323 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:54.213920 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:54.219343 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:54.409463 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:54.431030 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:54.438042 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:54.438627 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:54.438714 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:54.440291 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:54.452572 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:54.469259 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:54.544001 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:54.552999 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:54.555218 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:54.558085 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:54.559438 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:54.559849 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:54.579239 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:54.600734 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:54.604782 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:54.605360 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:54.612100 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:54.631167 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:54.635664 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:54.636804 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:54.717895 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:54.719237 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:54.719767 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:54.727190 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:54.730999 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:54.738723 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:54.751495 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:54.818838 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:54.836079 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:54.838573 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:54.840626 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:54.840708 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:54.843199 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:54.856223 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:54.864486 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:54.971874 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:54.976394 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:54.981241 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:54.987329 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:54.992108 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 68 similar messages]
I20240501 14:13:54.993347 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:54.996568 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 14:13:54.997207 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:55.002776 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:55.005947 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:55.006500 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240501 14:13:55.020421 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:55.021337 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:55.023207 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:55.029114 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:55.036446 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 14:13:55.046770 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:55.055330 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:55.104506 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 14:13:55.153899 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:55.155007 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:55.155684 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:55.159057 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:55.164685 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:13:55.164770 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 14:13:55.164901 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:55.176118 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 14:13:55.176311 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:55.182060 19356 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:55.182236 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:55.188822 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:55.200299 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:55.206669 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:55.210237 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:55.216827 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:55.225107 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:55.227679 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:55.240583 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:13:55.252074 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:13:55.368228 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:13:55.436937 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 14:13:55.522053 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:55.531587 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:55.538172 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:55.539151 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:55.550009 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:55.554715 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:55.565989 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:55.567438 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:55.570079 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 14:13:55.573135 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:55.598780 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:55.602419 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:55.609167 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:55.609826 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:55.612342 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:55.618822 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:13:55.632702 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:13:55.708806 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:55.716436 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:55.717929 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:55.718066 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:55.720571 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:55.723207 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:55.725955 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:55.729439 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:55.777797 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:55.786684 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:55.793359 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:55.793542 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:55.795735 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:55.796684 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:55.797706 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:55.900135 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:55.901108 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:55.902151 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:55.906774 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:55.908488 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:55.910540 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:55.934953 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:56.025945 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:56.039122 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:56.041186 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:56.041679 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:56.043368 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:56.044184 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:56.046563 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:56.115330 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:56.117784 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:56.120468 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:56.125329 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:56.127830 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:56.134994 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:56.143832 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:56.210963 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:56.212797 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:56.216154 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:56.219703 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:56.222200 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:56.223534 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:56.244745 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:56.252656 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:56.301487 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:56.302054 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:56.302862 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:56.303539 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:56.309177 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:56.315251 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:56.321310 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:56.512005 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:13:56.543215 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:56.543884 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:56.544775 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:56.548141 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:56.549146 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:56.549916 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:56.572093 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:56.612342 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 14:13:56.678478 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:56.679679 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:56.683732 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:56.687767 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:56.689776 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:56.694515 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:56.706677 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:56.728186 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:56.731451 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:56.732144 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:56.732029 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:56.741485 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:56.756361 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:56.760715 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:56.819540 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:56.852227 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:56.860335 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:56.861325 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:13:56.869029 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:56.870972 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:56.880378 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:56.889856 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:56.912240 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:56.918319 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:56.931569 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:56.934617 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:56.939193 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:56.940977 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:56.942946 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:56.959578 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:57.047809 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:57.049268 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:57.057622 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:57.058125 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:57.059585 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:57.075999 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:57.084692 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:57.162673 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:57.169268 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:57.169905 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:57.174468 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:57.189885 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:57.200119 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:57.200768 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:57.261813 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:57.262147 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:57.264379 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:57.267890 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:57.267953 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:57.281942 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:57.292309 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:57.369158 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:57.370653 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:57.371459 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:57.375662 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:57.377249 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:57.402395 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:57.402719 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:57.411688 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:13:57.412338 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:13:57.430631 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 14:13:57.504968 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:57.507866 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:57.511610 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:57.513072 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:57.515848 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:13:57.522572 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 14:13:57.526350 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:57.538302 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:57.545859 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:13:57.590595 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:13:57.596149 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 14:13:57.630365 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:57.630841 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:57.634157 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:57.637431 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:57.639204 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:57.640106 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:57.656534 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:57.691781 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:13:57.725320 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:13:57.726357 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 14:13:57.754765 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:57.760900 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:57.760994 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:57.774982 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:57.775256 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:57.775770 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 14:13:57.794222 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:57.803480 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:57.810129 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:57.812427 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:57.815908 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:57.816394 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:57.823210 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:57.835351 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:57.858016 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:13:57.888789 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:13:57.915037 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:13:57.915606 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:13:57.917779 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 14:13:57.966964 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:57.967882 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:57.977530 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:57.989193 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:57.990088 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:57.990818 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:57.994284 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:58.001673 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:58.004338 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:58.005069 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:58.009521 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:58.012079 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:58.033907 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:58.040282 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 14:13:58.054029 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:58.135798 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:58.136567 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:58.138707 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:58.140806 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:58.142695 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:58.151603 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:58.167236 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:58.192656 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 14:13:58.356549 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:58.356851 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:58.357753 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:58.359568 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:58.361488 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:58.362438 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:58.391081 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:58.477881 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:58.489363 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:58.491175 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:58.493861 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:58.500414 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:58.508123 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:58.521481 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:58.522022 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:58.538952 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:58.541364 21117 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:58.541385 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:58.547574 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:58.559046 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:58.571511 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:58.583446 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:58.671974 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:58.672858 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:58.674070 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:58.674088 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:58.685145 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:58.701668 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:58.707893 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:58.785353 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:58.789752 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:58.790189 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:58.799223 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:58.812646 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:58.813416 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:58.830107 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:58.907490 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:58.907713 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:58.912343 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:58.916687 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:58.921559 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:58.922143 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:58.934685 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:58.987449 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:58.993269 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:58.994511 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:58.996318 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:59.015229 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:59.018985 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:59.022612 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:59.032902 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:59.086038 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:59.088375 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:59.095255 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:59.097615 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:59.097981 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:59.120497 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:59.123332 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:59.154853 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 14:13:59.178577 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:59.178731 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:59.182130 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:59.192257 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:59.196321 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:59.196761 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:59.197762 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:59.417011 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:59.417894 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:59.420365 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:59.427784 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:59.428504 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:59.440968 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:59.452175 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:59.524502 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:59.524912 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:59.527030 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:59.533517 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:59.534286 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:59.536249 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:59.565707 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:59.570381 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:13:59.598026 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:59.600669 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:59.611500 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:59.611723 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:59.617565 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:59.634881 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:59.645661 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:59.711647 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:59.711723 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:59.716459 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:59.719816 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:59.721624 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:59.727984 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:59.742312 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:59.796653 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 14:13:59.824368 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:59.824812 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:59.828104 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:59.829938 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:59.838063 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:59.856859 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:59.862833 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:13:59.962977 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:59.963075 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:59.965286 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:59.969951 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:13:59.971975 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 14:13:59.982395 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:59.984371 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:59.989209 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:13:59.992573 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:13:59.999243 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:13:59.999528 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 14:14:00.002414 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:00.013615 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:00.016093 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:00.019802 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:14:00.019812 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 14:14:00.027902 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:00.029122 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:00.057329 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240501 14:14:00.141739 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:00.147217 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:00.147503 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:00.151185 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:00.151991 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:00.161011 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:00.169790 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 14:14:00.173485 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:00.183646 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 14:14:00.203565 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:00.203611 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:00.210276 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:00.216219 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:00.221349 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:00.227356 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:00.242527 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:00.326426 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:14:00.327744 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 14:14:00.328325 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:14:00.446431 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:14:00.462957 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:14:00.463009 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:14:00.478305 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 14:14:00.480542 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:00.486901 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:00.488998 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:00.489723 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:00.489936 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:00.490775 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:00.503263 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:00.522961 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:00.570590 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:00.571534 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:00.580713 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:00.582098 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:00.583782 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:00.597957 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:00.598448 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:00.644362 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 14:14:00.669804 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:00.669997 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:00.679001 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:00.699416 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:00.701584 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:00.705308 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:00.711277 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:00.762056 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:00.766007 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:00.766422 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:00.766876 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:00.766909 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:00.779270 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:00.796563 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:00.875028 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:00.876344 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:00.882347 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:00.887204 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:00.889246 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:00.900653 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:00.920732 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:01.016839 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:01.023423 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:01.027558 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:01.028765 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:01.030578 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:01.031517 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:01.039609 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:01.051187 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:01.076651 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:01.080993 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:01.085978 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:01.086637 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:01.090899 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:01.095465 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:01.108781 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:01.208549 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:01.211516 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:01.217008 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:01.218096 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:01.222283 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:01.223986 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:01.231498 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:01.238534 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:01.248224 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:01.249473 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:01.271566 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:01.273375 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:01.284070 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:01.288450 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:14:01.441054 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:14:01.513633 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 14:14:01.605993 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:01.607954 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:01.615348 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:01.619832 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:01.636829 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:01.641129 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:01.643220 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:01.645291 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:01.649403 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:01.652449 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:01.670895 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:01.680826 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:01.687549 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:01.714656 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:01.795547 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:01.801880 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:01.803934 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:01.813725 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:01.815816 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:01.823314 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:01.844338 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:01.860806 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:01.867761 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:01.869760 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:01.870812 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:01.870931 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:14:01.879743 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:14:01.889767 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:01.892042 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:01.906785 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:01.984583 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:01.986105 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:01.986624 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:01.993182 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:01.993314 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:02.005918 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:02.012745 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:02.024518 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:14:02.102247 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:02.104933 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:02.105511 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:02.109395 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:02.119473 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:02.137449 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:02.137511 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:02.178205 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 14:14:02.226272 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:02.234261 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:02.235994 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:02.236589 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:02.252543 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:02.262290 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:02.275672 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:02.275949 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:02.283226 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:14:02.289165 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 14:14:02.290957 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:02.305178 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:02.310478 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:02.312556 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:02.312927 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:02.370383 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 14:14:02.411705 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:02.412400 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:02.415323 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:02.416076 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:02.416924 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:02.423871 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:02.439954 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:02.485846 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:14:02.486554 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:14:02.497500 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 14:14:02.596518 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:02.597272 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:02.600001 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:02.600255 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:02.601783 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:02.605899 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:02.639019 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:02.666935 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 14:14:02.728487 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:02.737363 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:02.740981 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:02.742713 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:02.743970 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:02.752009 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:02.756791 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 14:14:02.780009 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:02.802033 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:02.808779 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:02.811978 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:02.816901 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 14:14:02.819715 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:02.823590 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:02.830494 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:02.831964 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:02.857627 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:14:02.879495 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:14:02.880337 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:14:02.906252 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 14:14:02.921649 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:02.922658 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:02.923404 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:02.923516 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:02.944548 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:02.962695 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:02.963002 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:02.968537 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:14:02.985379 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 14:14:02.996793 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:02.998606 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:02.998864 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:03.001080 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 14:14:03.013862 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:03.016588 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:03.020118 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:03.021368 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:03.108214 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:03.109027 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:03.109797 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:03.113001 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:03.129695 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:03.132164 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 14:14:03.136482 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:03.138490 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:03.247917 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:03.248281 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:03.251699 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:03.256537 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:03.267459 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:03.274255 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:03.276810 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:03.351166 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:03.352872 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:14:03.361784 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:03.362921 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:03.363747 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:03.369874 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:03.372282 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:03.399178 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:03.478586 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:03.479218 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:03.479929 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:03.481719 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:03.483491 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:03.491640 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:03.504467 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:03.583634 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:03.583971 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:03.585062 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:03.585095 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:03.594218 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:03.608930 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:03.619241 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:03.710258 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:03.713336 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:03.715451 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:03.718083 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:03.718888 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:03.724071 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:03.745427 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:14:03.811429 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 14:14:03.845825 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:03.849257 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:03.850399 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:03.855551 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:03.865871 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:03.871203 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:03.888348 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:03.902295 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:03.905664 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:03.909742 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:03.920524 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:03.921384 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:03.930292 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:03.942462 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:03.949821 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:04.038312 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:04.042663 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:04.047816 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:04.048610 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:04.068169 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:04.069964 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:04.072374 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:04.083168 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:04.107523 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:04.112044 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:04.115329 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:04.115969 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:04.117882 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:04.129099 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:04.222404 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:04.225090 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:04.226843 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:04.249042 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:04.249898 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:04.249989 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:04.259227 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:04.392827 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:04.410053 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:04.411509 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:04.414528 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:04.424151 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:04.433290 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:04.443439 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:04.448558 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:04.533484 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:04.534613 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:04.543574 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:04.544409 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:04.545302 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:04.567291 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:04.589529 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:04.598475 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:04.601928 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:04.604471 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:04.620200 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:04.620885 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:04.636128 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:04.638566 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:14:04.713647 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 14:14:04.725744 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:04.732988 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:04.734524 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:04.735680 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:04.739995 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:04.747211 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:04.748235 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:04.775341 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 14:14:04.851914 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:04.852399 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:04.852841 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:04.855683 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:04.856551 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:04.857192 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:04.883893 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:04.892520 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:14:04.910743 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:04.950526 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:04.950522 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:04.955392 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:04.964614 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:04.965540 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:04.979398 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:04.992117 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:04.997269 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:14:05.031210 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 14:14:05.047489 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:05.049201 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:05.051590 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:05.051834 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:05.054512 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:05.057861 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:05.064929 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:05.140609 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:05.141692 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:05.142956 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:05.143831 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:05.143913 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:05.163767 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:14:05.164513 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 14:14:05.166189 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 14:14:05.171252 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:05.218868 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:05.223209 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:05.226187 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:05.228658 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:05.229743 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:05.239985 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:05.260766 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:05.264591 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:14:05.428721 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:14:05.455905 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 14:14:05.471335 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:05.472920 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:05.483187 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:05.483901 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:05.485916 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 14:14:05.492511 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:05.493057 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:05.495826 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:05.562510 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 14:14:05.607547 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:05.618431 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:05.622556 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:05.626488 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:05.629880 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:05.633899 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:05.641945 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:05.647248 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:14:05.647403 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 14:14:05.649518 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:05.655372 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:05.656337 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:05.666821 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:14:05.668910 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 14:14:05.670374 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:14:05.683704 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 14:14:05.689739 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:05.700242 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:05.794991 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:05.797852 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:05.798883 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:05.803212 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:05.809533 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:05.816133 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:05.818238 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:05.871173 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:05.910135 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:05.912497 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:05.917383 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:05.918352 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:05.923395 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:05.925390 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:05.928499 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:06.032696 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 14:14:06.049108 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:06.052110 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:06.054561 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:06.064639 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:06.073752 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:06.074333 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:06.076748 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:06.082173 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:06.086364 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:06.086676 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:06.109593 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:06.117519 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:06.141955 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:06.145588 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:06.218550 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:06.222690 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:06.239065 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:06.248463 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:06.249495 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:06.254032 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:06.271854 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:06.272697 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:06.284971 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:06.297503 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:06.301905 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:06.306120 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:06.314250 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:06.314800 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:06.337333 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:06.622212 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:06.622956 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:06.628943 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:06.631266 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:06.634061 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:06.644937 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:06.655099 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:06.697141 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:06.698159 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:06.698681 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:06.706358 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:06.706735 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:06.709092 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:06.718827 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:06.719846 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:06.806488 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:06.807958 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:06.810693 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:06.812016 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:06.815810 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:06.825400 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:06.839702 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:06.884547 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:06.889146 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:06.890394 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:06.896986 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:06.898690 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:06.907316 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:06.908973 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:14:06.931990 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:14:07.000363 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:07.001048 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:07.002502 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:07.003796 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:07.011978 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:07.012665 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:07.015775 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:07.124405 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:07.127964 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:07.129242 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:07.142942 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:07.144018 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:07.145677 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:07.147519 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:07.184547 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 14:14:07.188603 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:07.196298 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:07.197660 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:14:07.197737 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 14:14:07.198168 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:07.203505 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:07.212756 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 14:14:07.219066 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:07.221278 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:14:07.232347 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:07.245637 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:07.320950 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:07.322041 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:07.340525 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:07.340544 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:07.341910 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:07.342842 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:07.348510 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:07.354365 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:07.356283 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:07.377609 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:07.385285 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:07.386088 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:07.394593 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 14:14:07.395812 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:07.400228 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:14:07.436861 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:14:07.527520 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:14:07.597357 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:14:07.649003 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:14:07.656669 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 14:14:07.721446 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:07.724845 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:07.745735 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:07.751742 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:07.758307 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:07.761121 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:07.761694 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:07.762658 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:07.763031 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:07.794891 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:07.799275 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:07.802913 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:07.802995 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:07.820115 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:07.883803 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:14:07.909180 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 14:14:07.925968 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:07.926697 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:07.929664 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:07.937664 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:07.953755 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:07.955974 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:07.960712 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:07.991453 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:07.999608 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:08.002839 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:08.002839 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:08.015668 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:08.017789 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:08.032728 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:08.102591 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:14:08.114902 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:08.115499 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:08.117863 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:08.118461 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:08.126418 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:08.135190 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:08.143302 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:08.162418 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:14:08.207600 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:08.232856 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:08.233064 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:08.233633 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:08.236574 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 14:14:08.242502 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:08.247170 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:08.249168 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:08.282267 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:08.323184 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:08.325368 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:14:08.330371 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 14:14:08.334106 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:08.334779 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:08.339376 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:08.348140 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:08.369654 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:08.416215 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:08.416606 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:08.420907 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:08.435416 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:08.435559 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:08.437005 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:08.451016 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:08.502143 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:08.508324 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:08.510116 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:08.511772 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:08.512200 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:08.527755 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:08.537837 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:08.585598 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:14:08.646217 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:08.739804 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:08.741897 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:08.749373 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:08.749979 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:08.750861 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:08.754622 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:08.769927 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:08.871171 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:08.871732 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:08.872879 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:08.883306 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:08.896384 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:08.903223 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:08.907972 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:08.930588 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:08.931205 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:08.931807 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:08.937783 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:08.947780 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:08.950475 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:08.953150 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:08.955157 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:14:09.040266 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:09.052561 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:09.055750 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:09.061389 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:09.070346 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:09.078356 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:09.081243 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:09.091925 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:09.103179 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:09.103240 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:09.110294 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:09.111173 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:09.119733 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:09.131683 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:09.147279 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:09.227548 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:09.229941 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:09.234812 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:09.244760 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:09.247810 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:09.257524 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:09.273496 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:09.343873 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:09.344167 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:09.349117 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:09.349201 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:09.351969 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:09.373165 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:09.374212 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:09.450515 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:09.451799 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:09.453163 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:09.454384 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:09.462085 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:09.468214 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:09.478928 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:09.556406 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:09.556890 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:09.559469 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 14:14:09.563453 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:09.564888 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:09.565188 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:14:09.575037 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:09.577756 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:09.581288 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:09.664229 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:09.666128 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:09.672590 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:09.672816 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:09.679836 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:09.682145 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 14:14:09.682579 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:09.682961 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:09.696205 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:14:09.712191 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 14:14:09.779553 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:09.780819 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:09.787698 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:09.789692 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:09.797533 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:09.823604 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:09.832715 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:09.892699 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:14:09.894574 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 14:14:09.902937 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:09.911526 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:09.912426 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:09.922407 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:09.924860 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:09.952590 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:09.954344 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:09.975291 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:09.983218 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:09.983661 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:09.988001 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:09.989678 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:09.990391 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:09.995194 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:10.075884 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:10.083678 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:10.083899 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:10.084415 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:10.085275 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:10.092165 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:10.103333 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:10.113925 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:10.161177 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:10.161592 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:10.164858 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:10.166482 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:10.173642 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:10.174836 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:10.175732 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 14:14:10.198997 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:10.213521 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 14:14:10.214159 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:14:10.248075 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 14:14:10.278470 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:10.280128 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:10.280721 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:10.282060 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:10.285595 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:10.295394 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:10.298543 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:10.411871 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:10.413475 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:10.414122 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:10.422029 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:10.433564 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:10.435473 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:10.437119 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:10.524037 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:10.527967 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:10.528189 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 14:14:10.531324 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:10.532827 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:10.533993 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:10.534330 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:10.551551 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:10.625838 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:10.627463 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:10.629129 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:10.638480 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:10.639026 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:10.640877 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:10.650635 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:10.677345 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:14:10.684690 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:14:10.710443 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 14:14:10.735960 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:10.736908 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:10.740944 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:10.747558 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:10.749629 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:10.753482 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:10.769920 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 14:14:10.774803 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:10.871649 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:10.872145 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:10.876144 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:10.884009 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:10.884352 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:10.888656 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:10.913024 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:10.913640 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:10.914040 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:10.921343 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:10.922295 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:10.935187 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:10.939224 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:10.941562 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:11.052553 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:11.057654 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:11.067255 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:11.070896 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:11.071591 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:11.071779 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:11.083448 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:11.106971 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:11.107012 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:11.113054 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:11.129319 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:11.129480 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:11.136209 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:11.137497 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:11.182848 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 14:14:11.224287 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:11.228808 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:11.230319 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:11.231516 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:11.239128 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:11.255911 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:11.266461 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:11.289309 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:11.388171 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:11.393606 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:11.396080 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:11.396844 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:11.404482 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:11.418445 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:11.434916 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:11.505452 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:11.511430 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:11.515951 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:11.526084 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:11.529724 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:11.535300 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:11.551728 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:11.590039 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:11.590600 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:11.595253 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:11.598397 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:11.602311 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 14:14:11.602536 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:11.610512 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:11.627697 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:11.712307 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:11.712397 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:11.713204 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:11.722301 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:11.723281 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:11.735421 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:11.735616 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:14:11.791316 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:11.829808 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:11.833009 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:11.840517 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:11.842837 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:11.843511 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:11.855201 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:11.875182 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:11.901767 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:11.903406 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:11.906586 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:11.906849 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:11.910593 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:11.911345 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:11.932288 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:11.950407 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:14:12.031450 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:12.033242 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:12.035887 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:12.037429 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:12.039196 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:12.049515 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:12.058398 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:12.069427 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:12.079968 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:12.083855 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:12.084671 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:12.089625 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:12.100353 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:12.103749 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:12.157069 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:14:12.185058 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:12.196583 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:12.196703 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:12.197646 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:12.203917 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:12.204200 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:12.207458 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:12.220741 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:14:12.265839 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:14:12.270974 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:14:12.322216 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:14:12.366576 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 14:14:12.398948 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:12.401219 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:12.402724 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:12.412597 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:12.424264 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:12.425042 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:12.431265 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:12.459343 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:14:12.471462 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 14:14:12.523864 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:12.530622 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:12.531076 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:12.548450 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:12.554304 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:12.555375 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:12.556075 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:12.560112 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:12.563213 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:12.572897 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:12.584440 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:12.587603 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:12.613634 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:12.615397 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:12.693653 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:12.697150 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:12.697674 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:12.701444 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:14:12.702471 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:14:12.702929 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:12.708272 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:12.709136 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 14:14:12.731519 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:12.748379 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:14:12.779618 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 14:14:12.810982 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:12.812961 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:12.813661 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:12.814312 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:12.825122 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:12.827231 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:12.831946 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:12.836345 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:14:12.872102 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 14:14:12.932360 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:12.933099 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:12.935359 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:12.940325 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:12.940975 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:12.947245 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:12.963966 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:12.999131 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:12.999673 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:13.007792 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:13.012513 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:13.013159 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:13.027344 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:13.035989 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:13.111671 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:13.123483 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:13.125293 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:13.129199 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:13.137149 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:13.144075 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:13.149600 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:13.165300 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:13.167245 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:13.177270 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:13.186389 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:13.188578 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:13.190840 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:13.203636 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:13.203814 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 14:14:13.244405 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:14:13.274153 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:14:13.362149 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:13.503813 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:13.509102 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:13.513008 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:13.519742 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:13.521574 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:13.525799 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:13.535471 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:13.540477 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:13.546205 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:13.547634 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:13.548413 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:13.563370 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:13.566213 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:13.579895 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:13.660439 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 14:14:13.681121 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:13.690232 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:13.693743 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:13.694578 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:13.695732 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:13.696578 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:13.715158 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:13.725395 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:13.726727 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:13.728622 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:13.742719 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:13.767205 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:13.781019 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:13.781445 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:13.869376 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:13.870666 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:13.873572 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:13.876631 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:13.883904 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:13.885852 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:13.903379 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:13.962355 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:13.987077 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:13.988472 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:13.988762 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:13.992874 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:14.013442 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:14.023634 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:14.030581 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:14.095439 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:14.104779 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:14.105369 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:14.105572 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:14.107515 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:14.115751 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:14.116650 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:14.171393 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:14.176693 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:14.180008 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:14.184875 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:14.190708 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:14.203120 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:14.205636 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:14.239919 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 14:14:14.276079 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:14.278517 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:14.279613 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:14.281450 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:14.289263 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:14.289891 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:14.292999 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:14.374617 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:14.498059 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:14.499310 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:14.501786 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:14.507395 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:14.512888 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:14.525014 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:14.525173 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:14.611953 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:14.611960 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:14.622161 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:14.633246 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:14.638114 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:14.643543 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:14.646322 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:14.658319 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 14:14:14.678815 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:14.689152 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:14.689616 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:14.695850 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:14.700166 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:14.707186 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:14.710309 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:14.779093 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:14:14.796310 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:14.817169 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:14.819065 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:14.823323 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:14.826783 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:14.842953 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:14:14.848042 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 14:14:14.850872 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:14.851995 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:14.861979 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:14.871034 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:14.873013 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:14.875762 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:14.898506 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:14.905416 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:14.927737 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:15.006780 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 14:14:15.009183 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:15.009665 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:15.014498 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:15.015425 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:15.018819 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:15.022462 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:15.029549 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 14:14:15.038085 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:15.043293 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:14:15.059096 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 14:14:15.129318 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:15.134936 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:15.135061 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:15.136679 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:15.138038 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:15.145444 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:15.177752 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:15.184350 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:14:15.224081 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 14:14:15.224234 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:15.228904 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:15.233968 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:15.239091 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:15.243162 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:15.244818 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:15.255939 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:15.274022 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:15.318184 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 14:14:15.352555 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:15.354111 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:15.357736 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:15.359217 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:14:15.361858 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 14:14:15.366870 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:15.377897 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:15.385824 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:15.466567 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:15.467132 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:15.470255 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:15.473294 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:15.474993 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:15.487915 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:15.490105 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:15.577601 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:15.578487 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:15.581892 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:15.584242 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:15.584429 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:15.596982 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:15.620049 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:15.643661 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 14:14:15.692974 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:15.693305 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:15.699085 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:15.707366 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:15.708962 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:15.714052 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:15.717700 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:15.723729 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:15.778676 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:15.781025 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:15.787765 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:15.795666 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:15.796865 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:15.800184 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:15.816252 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:15.825349 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 14:14:15.878957 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:15.879364 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:15.883157 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:15.886807 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:15.889467 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:15.900444 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:15.906018 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:15.907616 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 14:14:15.965812 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:15.969373 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:15.970619 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:15.974679 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:15.983381 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:15.986249 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:16.004799 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:16.084674 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:16.085182 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:16.089395 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:16.096100 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:16.106881 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:16.107034 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:16.107698 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:16.141726 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:16.210984 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:16.217132 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:16.218130 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:16.218544 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:16.220432 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:16.228188 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:16.253082 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:16.327852 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:16.329352 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:16.330884 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:16.334146 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:16.340760 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 14:14:16.341290 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:16.342268 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:16.345224 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:16.435707 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:16.437317 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:16.438951 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:16.447029 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:16.450996 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:16.451260 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:16.473774 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:16.557583 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:16.557786 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:16.559445 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:16.567770 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:16.568399 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:16.571940 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:16.587690 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:16.675956 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:16.675966 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:16.682507 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:16.683562 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:16.691864 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:16.700837 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 14:14:16.704779 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:16.713999 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:16.714085 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:16.715554 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:16.722756 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:16.728385 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:16.733889 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:16.744302 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:16.755617 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:16.757141 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:16.849817 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:16.854744 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:16.865322 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:16.867570 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:16.870055 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:16.870538 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:16.892448 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:16.915829 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 14:14:16.933688 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:16.935719 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:16.936103 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:16.940778 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:16.941324 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:16.944218 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:16.964435 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:17.042773 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:17.043149 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:17.051640 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:17.052021 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:17.055207 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:17.059548 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:17.089149 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:17.189724 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:17.190829 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:17.193215 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:17.203150 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:17.204906 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:17.206912 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:17.208580 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:14:17.242022 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:14:17.282148 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 14:14:17.306514 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:17.318477 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:17.318759 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:17.318802 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:17.319722 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:17.319792 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:17.320434 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:17.395139 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:14:17.401206 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 14:14:17.421353 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:17.422163 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:17.422407 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:17.422981 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240501 14:14:17.425954 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:17.426761 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:17.453466 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:14:17.454442 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 14:14:17.455574 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:17.455659 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:17.478317 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 14:14:17.538972 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:17.541301 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:17.548389 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:17.548540 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:17.552259 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:17.562575 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:17.565233 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:17.637310 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:17.664599 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:17.670217 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:17.671705 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:17.672173 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 14:14:17.676043 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:17.699529 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:17.705991 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:17.709038 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:17.711601 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:17.721014 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:17.726197 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:17.727242 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:17.737452 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:17.739846 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 14:14:17.740805 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:17.757402 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:14:17.764914 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 14:14:17.769862 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:17.848295 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:17.850327 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:17.852262 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:17.853914 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:17.859236 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:17.862758 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:17.893986 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:17.931237 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:17.932888 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:17.934450 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:17.934845 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:17.938429 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:17.944229 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:17.966753 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:18.047878 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:14:18.049153 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 14:14:18.050009 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:18.050824 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:18.055616 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:18.056327 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:18.063562 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:18.068181 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:18.081897 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:18.146800 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:18.205451 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:18.207252 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:14:18.209723 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:14:18.210503 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:18.217995 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:18.222306 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:18.222623 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:18.237869 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:18.289064 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 14:14:18.325242 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:18.326112 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:18.327728 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:18.337057 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:18.351578 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:18.355691 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:18.368362 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:18.404469 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:18.412586 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:18.413062 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:18.417446 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:18.418107 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:18.422457 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:18.435732 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:18.536584 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:18.537014 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:18.537668 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:18.546077 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:18.548251 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:18.552599 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:18.557001 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:18.648682 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:18.658939 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:18.660107 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:18.664296 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:18.668182 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:18.670331 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:18.675269 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:18.694942 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:18.703182 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:18.713367 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:14:18.713577 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 14:14:18.723189 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:18.729758 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:18.733235 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:18.738543 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:18.745257 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:18.840061 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:18.849740 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:18.851258 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:18.852250 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:18.852887 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:18.855317 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:18.856163 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:18.864161 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:18.885633 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:18.887512 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:18.898706 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:18.906611 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:18.908730 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:18.927210 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:19.013334 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:19.018198 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:19.022841 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:19.026499 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:19.033325 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:19.036566 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:19.037086 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:19.122033 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:14:19.218147 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:19.227886 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:19.229919 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:19.230381 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:19.230872 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:19.231889 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:19.251523 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:19.263058 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:19.357964 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:19.358568 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:19.362213 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:19.370477 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:19.372426 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:19.378723 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:19.383450 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:14:19.387643 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 14:14:19.398355 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:19.409999 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:19.411195 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:19.411724 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:19.413326 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:19.422109 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:19.453465 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:19.533694 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:19.534343 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:19.534430 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:19.550027 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:19.550338 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:19.556641 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:19.557356 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:19.594583 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:14:19.602092 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:19.645375 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:19.647862 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:19.658213 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:19.659499 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:19.665205 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:19.669137 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:19.678040 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 14:14:19.692320 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:19.768489 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:19.770093 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:19.770666 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:19.775343 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:19.783699 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:19.791468 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:19.821797 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:19.822198 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 14:14:19.836935 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:19.838493 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:19.842031 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:19.851080 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:19.857939 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:19.858680 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:19.863633 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:14:19.876762 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:14:19.877084 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 14:14:19.897323 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:19.963222 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:19.964536 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:19.966332 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:19.968107 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:19.981456 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:19.987726 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:19.987977 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:20.017400 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:20.022703 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:20.038180 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:20.044943 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:20.047194 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:20.060916 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:20.068012 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 14:14:20.074450 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:14:20.095326 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:14:20.168098 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:14:20.270165 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 14:14:20.271682 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 14:14:20.354172 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:20.361522 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:20.366694 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:20.367496 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:20.369084 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:20.372797 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:20.383221 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:20.404160 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 14:14:20.425305 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:20.426127 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:20.434832 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:20.442001 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:20.442009 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:20.442615 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:20.462376 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:20.537917 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:20.539275 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:20.544536 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:20.544921 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:20.550073 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:20.551719 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:20.566447 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:20.594491 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:20.598508 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:20.603750 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:20.612841 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:20.617622 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:20.618606 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:20.626904 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:20.635959 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:20.722807 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:20.725162 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:20.726089 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:20.726156 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:20.730899 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:20.734174 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:20.741190 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:20.837513 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:20.840564 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:20.842082 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:20.843078 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 14:14:20.843686 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:20.849556 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:20.851420 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:20.874858 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:20.960877 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:20.963101 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:20.966598 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:20.979554 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:20.993532 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:20.993549 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:21.004871 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:21.005228 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:21.006832 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:21.012347 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:21.032130 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:21.038867 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:21.042060 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:21.058358 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:21.075901 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:14:21.123121 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240501 14:14:21.132230 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:21.133121 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:21.141639 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:21.141705 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:21.147315 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:21.150108 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:21.164320 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:14:21.246757 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 14:14:21.344767 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:21.345480 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:21.345953 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:21.352079 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:21.364631 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:21.364797 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:21.369060 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:21.459908 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:21.460098 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:21.463416 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:21.470451 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:21.478205 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:21.488170 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:21.489415 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:21.526872 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:21.529560 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:21.533316 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:21.536787 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:21.537330 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:21.558465 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:21.558832 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:21.560266 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:21.622159 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:21.623373 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:21.625746 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:21.626703 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:21.639586 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:21.653991 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:21.660923 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:21.675762 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 14:14:21.711226 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:21.714998 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:21.715232 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:21.715281 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:21.717820 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:21.742276 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:21.743917 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:21.802870 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 14:14:21.832199 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:21.833199 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:21.834740 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:21.838984 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:21.857314 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:21.862192 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:21.865095 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:21.964747 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 14:14:21.970019 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:21.977620 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:21.981284 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:21.982362 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:21.986131 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:21.987247 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:21.988546 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:21.993229 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:21.997298 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:22.003587 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:22.012588 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 14:14:22.014464 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:22.014624 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:22.016513 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:22.023398 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:22.073405 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:22.127672 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 14:14:22.137789 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:22.138092 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:22.139643 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:22.144441 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:22.151932 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:22.164002 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:22.164829 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:22.274209 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 14:14:22.352440 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:22.354632 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:22.358009 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:22.363335 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:22.366381 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:22.377764 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:22.378484 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:22.470643 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:22.474049 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:22.476692 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:22.479097 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:14:22.479103 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 14:14:22.480854 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:22.488252 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:22.494153 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:14:22.501163 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:22.515681 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:22.517838 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:22.549296 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:22.549788 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:14:22.553671 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:22.558063 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:14:22.558259 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 14:14:22.563112 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:22.564059 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:22.566735 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:22.571235 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:22.643744 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:22.648021 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:22.653270 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:22.663540 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:22.667977 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:22.670732 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:22.683531 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:22.729770 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:22.731457 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:22.735373 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:22.738477 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:22.741564 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:22.742149 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:22.750403 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:22.803509 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 14:14:22.846432 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:22.847252 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:22.857573 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:22.859191 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:22.863232 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:22.863835 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:22.887408 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:22.920888 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 14:14:22.972940 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:22.974156 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:22.978252 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:22.983352 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:22.987146 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:22.993419 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:23.011330 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:23.020229 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 14:14:23.052778 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:23.057570 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:23.059659 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:23.061002 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:23.071275 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:23.071579 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:23.073653 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:14:23.111106 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:23.168423 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:23.169083 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:23.172868 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:23.176782 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:23.179731 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:23.179998 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:23.201975 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:23.321457 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 14:14:23.322571 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:23.322914 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:23.326027 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:14:23.327241 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:23.334892 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:23.337450 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:23.337808 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:23.338119 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:14:23.394110 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 14:14:23.461164 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:23.466866 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:23.469537 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:23.469939 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:23.471125 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:23.489415 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:23.498924 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:23.523057 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:14:23.548833 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:23.549243 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:23.558157 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:23.559405 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:23.559473 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:23.563330 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:23.571552 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:23.637567 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:23.641129 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:23.649416 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:23.649999 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:23.651772 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:23.652406 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:23.652462 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:23.667526 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:23.745322 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:23.746783 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:23.750720 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:23.750762 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:23.760882 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:23.761598 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:23.767644 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:23.826500 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:23.827270 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:23.828480 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:23.830056 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:23.831831 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:23.846366 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:23.851555 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:23.932020 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 14:14:23.936170 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:23.939595 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:23.944039 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:23.945462 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:23.948658 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:23.952000 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:23.956761 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:24.072254 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:24.072564 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:24.075623 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:24.076980 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:24.083984 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:24.097143 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:24.099442 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:24.187278 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:24.189586 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:24.195432 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:24.197862 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:24.199077 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:24.199223 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:24.206735 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:24.219547 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:24.302660 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:24.304056 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:24.308140 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:24.312582 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:24.316792 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:24.321147 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:24.327324 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:24.373566 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 14:14:24.414824 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:24.419874 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:24.421311 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:14:24.431728 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:14:24.434015 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 14:14:24.434299 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:24.435313 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:24.440408 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:24.442821 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:24.472736 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 14:14:24.530773 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:24.530962 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:24.532419 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:24.532949 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:24.540846 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:24.559286 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:24.561295 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:24.569540 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:24.582064 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:24.592456 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:24.603714 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:24.611004 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:24.613870 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:24.613979 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:24.641786 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 14:14:24.713599 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:24.719024 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:24.721453 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:24.724893 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:24.738708 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:24.740262 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:24.752090 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:24.752745 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:24.762382 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:24.763099 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:24.764297 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:14:24.765014 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 14:14:24.781559 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:24.791384 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 14:14:24.798185 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:24.800134 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:24.811862 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:24.893499 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:24.896303 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:24.896771 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:24.896989 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:24.910195 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:24.927263 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:24.928350 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:25.089025 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:25.092147 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:25.092667 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:25.094799 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:25.096459 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:25.104316 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:25.113988 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:25.158270 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:14:25.193989 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 14:14:25.208667 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:25.209040 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:25.209733 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:25.218261 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:25.219504 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:25.233167 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:25.257333 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:25.268723 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:25.269397 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:25.270236 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:25.275550 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:25.286590 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240501 14:14:25.287253 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:25.291532 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:25.315240 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:25.333141 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:14:25.365921 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:25.401563 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:25.402172 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:25.403987 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:25.404027 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:25.408357 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:25.412016 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:25.438199 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:25.502290 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 14:14:25.513931 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:25.514591 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:25.515599 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:25.522610 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:25.529839 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:25.540956 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:25.553884 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:25.633445 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:25.633824 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:25.636039 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:25.638298 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:25.652473 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:25.655030 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:25.656503 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:25.709744 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:25.714692 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:25.715423 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:25.718400 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:25.727594 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:25.729276 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:25.749253 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:25.810272 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:25.813079 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:25.814137 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:25.814496 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:25.828308 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:25.841323 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:25.841621 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:25.894038 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:25.895802 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:25.899893 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:25.900730 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:25.905071 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:25.905385 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:25.911468 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:25.945385 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:14:26.089655 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:14:26.123632 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 14:14:26.143754 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:26.144744 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:26.147378 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:26.147861 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:26.151808 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:26.153200 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:26.173352 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:26.175370 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 14:14:26.264052 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:26.267627 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:26.274255 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:26.278218 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:26.280452 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:26.295019 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:26.317595 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:26.323802 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:26.330410 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:26.330603 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:26.333449 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:26.340688 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:26.351749 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:26.351822 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:26.361763 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 14:14:26.438146 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:26.440805 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:26.442170 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:26.451011 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:26.462762 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:26.463757 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:26.474174 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:26.476691 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:26.556643 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:26.557015 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:26.561143 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:26.561507 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:26.571961 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:26.577400 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:26.584044 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:26.682334 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:26.683189 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:14:26.694564 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 14:14:26.699250 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:26.701597 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:26.705686 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:26.723315 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:26.734896 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:26.748178 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:26.752434 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:26.752938 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:26.754323 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:26.762643 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:26.764860 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:26.775676 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:26.801679 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:14:26.841892 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 14:14:26.885076 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:26.885864 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:26.888885 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:26.889209 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:26.900760 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:26.901612 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:26.907840 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:26.925179 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:26.926797 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:26.942314 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:14:26.949198 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 14:14:26.951696 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:26.956329 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:14:26.956943 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 14:14:26.968513 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:26.973248 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:14:27.068951 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:14:27.257519 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 14:14:27.272372 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:27.278043 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:27.284269 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:27.286365 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:27.294349 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:27.304540 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:27.314925 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:27.349493 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:27.351119 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:27.351667 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:27.352900 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:27.361783 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:27.362098 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:27.371421 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 14:14:27.376149 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:27.459462 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:27.462316 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:27.462514 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:27.465415 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:27.472004 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:27.472254 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:27.514153 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:27.516886 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:27.517732 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:27.525930 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:27.531730 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:27.534300 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:27.534435 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:14:27.536427 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 14:14:27.555214 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:27.645887 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:27.647730 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:27.649176 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:27.656174 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:27.656451 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:27.667091 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:27.669593 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:27.678017 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:27.727551 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:14:27.742480 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 14:14:27.767028 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:27.767339 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:27.779497 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:27.781503 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:27.781839 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:27.781916 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:27.815914 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:27.840335 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:14:27.874048 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:27.877806 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:27.878203 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:27.883080 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:27.893929 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 14:14:27.899688 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:27.900802 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:27.914139 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:27.962976 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:27.963382 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:27.970700 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:27.977654 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:27.977955 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:27.980979 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:27.999290 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:28.062093 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:28.062175 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:28.069367 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:28.073242 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:28.074874 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:28.076004 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:28.077839 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:28.127504 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 14:14:28.149909 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:28.291396 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:28.293980 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:28.298076 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:28.302111 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:28.307222 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:28.318928 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:28.319058 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:28.399276 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:28.406862 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:28.408947 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:28.412206 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:28.420326 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:28.439003 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:28.439690 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:28.473603 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:28.481030 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:28.481240 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:28.486063 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:28.488157 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:28.491722 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:28.497481 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:28.557577 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 14:14:28.610527 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:28.612499 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:28.613574 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:28.615206 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:14:28.617838 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:14:28.623183 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:28.624450 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:28.632946 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:28.637231 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:28.651154 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:28.655845 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:28.655977 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:28.665940 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:28.669214 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:28.672067 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:28.692973 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:28.786603 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:28.800880 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:28.803537 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:28.803848 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:14:28.804247 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:28.810438 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:28.823196 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:28.833179 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:28.850039 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:14:28.887800 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 14:14:28.920630 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:28.922226 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:28.924852 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:28.930567 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:28.931850 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:28.938663 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:28.952770 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:29.022475 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:29.028267 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:29.033476 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:29.044181 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:29.049665 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:29.055193 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:29.055269 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:29.112159 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:29.143620 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:29.144296 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:29.145085 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:29.151211 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:29.151624 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:29.175383 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:29.180065 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:29.183720 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 14:14:29.250270 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:29.254406 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:29.258853 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:29.260458 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:29.269842 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:29.276751 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 14:14:29.277980 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:29.278134 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:14:29.340535 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 14:14:29.367295 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:29.368363 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:29.369031 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:29.372082 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:29.384964 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:29.390894 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:29.409786 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:29.454821 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 14:14:29.477561 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:29.477630 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:29.482698 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:29.490979 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:29.492911 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:29.494139 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:29.513485 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:14:29.516181 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:29.549263 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:29.550747 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:29.557897 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:29.561472 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:29.564493 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:29.579576 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 14:14:29.585916 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:29.591715 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:29.597432 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 14:14:29.649874 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:29.650907 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:29.651585 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:29.656261 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:29.665937 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:29.675583 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:29.684110 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:29.717123 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 14:14:29.749644 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:29.751750 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:29.752707 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:29.756374 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:29.766464 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:29.768158 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:29.770056 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:29.864147 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:29.864946 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:29.865689 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:29.866103 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:29.869813 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:29.869863 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:29.870800 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:29.900378 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:29.983901 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:29.991623 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:29.995857 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:30.004315 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:30.008721 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:30.010628 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:30.029235 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:30.117872 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:30.121294 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:30.121778 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:30.122323 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:30.124106 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:30.124143 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:30.152182 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:30.217564 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:14:30.219885 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 14:14:30.220283 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:30.225847 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:30.226256 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:30.226629 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:30.228274 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:30.228986 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:14:30.315443 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:14:30.321861 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:14:30.338820 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240501 14:14:30.339617 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:30.340308 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:30.341264 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 14:14:30.341691 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:30.342433 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:30.342440 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:30.366984 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:30.367574 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:30.371583 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:30.452522 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:30.458819 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:30.460939 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:30.481763 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:30.482266 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:30.483924 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:30.508176 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:30.509698 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:30.510279 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:30.511204 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:30.518137 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:30.524353 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:30.537093 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:30.547806 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:30.627918 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:30.630771 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:30.633273 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:30.641968 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:30.643236 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:30.664116 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:30.677198 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:30.681649 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:30.684546 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:30.697925 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:30.698547 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:30.698616 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:30.708015 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:30.713166 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:30.794888 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:30.811949 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:30.812505 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:30.812965 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:30.821319 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:30.824450 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:30.831825 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:30.845697 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:30.996423 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:30.996984 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:30.997720 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:31.000540 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:31.002393 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:31.011765 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:31.016176 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:31.110285 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:31.113349 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:31.114907 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:31.117475 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:31.128170 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:31.136538 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:31.144966 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:31.183442 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:31.185120 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:31.191788 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:31.192111 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:31.194146 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:31.204111 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:31.208534 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:31.288506 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 14:14:31.301153 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:31.302556 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:31.303632 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:31.312358 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 14:14:31.313782 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:31.319700 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:31.319706 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:31.333230 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:31.343876 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 14:14:31.345255 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:14:31.408396 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:31.419271 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:31.422745 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:31.424753 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:31.434224 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:31.437443 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:31.445526 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:31.459533 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:31.497644 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:31.498322 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:31.502156 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:31.503755 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:31.511171 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:31.516743 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:31.533613 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:31.590224 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:31.592226 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:31.595889 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:31.601495 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:31.601796 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:31.620314 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:31.621518 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:31.640060 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 14:14:31.679196 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:31.681344 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:31.684900 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:31.701453 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:31.713443 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:31.714793 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:31.726810 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:31.776289 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:31.777444 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:31.778419 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:31.783231 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:31.795296 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:31.798394 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:31.812348 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:31.845695 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:14:31.934365 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:14:31.977515 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 14:14:32.022182 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:32.024999 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:32.031685 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:32.036798 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:32.037060 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:32.041039 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:14:32.059619 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 14:14:32.072582 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:32.127835 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:32.130045 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:32.133031 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:32.135169 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:32.148375 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 14:14:32.156404 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:32.157255 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:32.176605 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:32.197232 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:32.197302 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:32.200239 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:32.220777 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:32.220780 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 14:14:32.222069 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:32.222697 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:32.223866 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:32.268074 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:32.297936 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:14:32.298091 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 14:14:32.299741 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:32.301797 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:32.304767 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:32.308192 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:32.309690 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:32.322098 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:32.411438 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:32.411581 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:32.412598 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:32.413129 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:32.413584 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:32.431963 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:32.441519 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:32.442687 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 14:14:32.547492 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:32.548280 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:32.551548 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:32.557126 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:32.557657 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:32.560423 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:32.573524 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:32.574223 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:32.605837 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:32.613992 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:32.614336 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:32.617491 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:32.621775 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:32.623481 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:32.666774 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:32.718593 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:32.722252 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:32.723728 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:32.725898 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:32.736635 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:32.738224 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:32.759270 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:32.783622 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 14:14:32.785444 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:32.790390 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:32.796829 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:32.804916 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:32.807730 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:32.819619 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:32.832854 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:32.840927 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:14:32.904495 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:14:33.051945 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:14:33.080745 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:33.083088 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:33.093132 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:33.109907 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:33.110289 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:33.110476 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:33.114780 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:33.114959 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:33.163368 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:33.166277 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:33.174952 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:33.175808 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:33.184126 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:33.195240 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:33.197417 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:14:33.211172 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 32 similar messages]
I20240501 14:14:33.272035 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:33.276846 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:33.280138 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:33.282112 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:33.283241 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:33.295583 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:33.309659 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:33.339807 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:33.344118 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:33.347175 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:33.355973 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:33.356257 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:33.364611 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:33.374976 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:33.463791 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:33.467172 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:33.467200 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:33.467350 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:33.471047 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:33.487332 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:33.496392 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:33.586396 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:33.588152 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:33.594962 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:33.595016 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:33.595386 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:33.601162 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:33.602146 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:33.621495 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:33.696255 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:33.698735 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:33.705835 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:33.715189 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:33.731971 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:14:33.753119 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:33.754738 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:33.755867 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 14:14:33.756320 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:33.762239 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:33.763002 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:33.767045 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:33.767540 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:33.773384 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:33.793196 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:33.809600 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:33.888152 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:33.888391 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:33.891718 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:33.897819 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:33.898420 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:33.905934 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:14:33.927542 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 14:14:33.928704 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:33.933696 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:14:33.970288 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:14:34.094249 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:34.099107 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:34.099895 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:34.106890 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:34.117360 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:34.118479 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:34.119824 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:34.143599 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:14:34.163995 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:14:34.190104 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 14:14:34.226037 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:34.229456 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:34.231223 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:34.237707 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:34.246166 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:34.246444 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:34.246446 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:34.248420 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:34.256855 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:34.272995 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:34.278597 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:34.284389 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:34.285774 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:34.310595 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:34.413583 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:34.416695 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:34.419284 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:34.419641 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:34.421947 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:34.429049 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:34.433077 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:34.441563 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:34.449606 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:34.449648 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:34.456918 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:34.468055 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:34.471330 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:34.492323 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:14:34.543411 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 14:14:34.580973 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:34.581885 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:34.582356 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:34.582703 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:14:34.590862 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 14:14:34.592768 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:34.607740 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:34.608767 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:34.617394 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:14:34.630174 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:14:34.651814 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 14:14:34.696456 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:34.698385 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:34.699132 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:34.699445 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:34.699579 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:34.711800 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:34.711880 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:14:34.746325 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 14:14:34.804075 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:34.804363 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:34.807924 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:34.808789 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:34.810055 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:34.819599 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:34.825879 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:34.832612 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 14:14:34.923736 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:34.923842 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:34.925266 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:34.930779 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:34.932672 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:34.939988 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:34.953039 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:35.029402 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:35.038646 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:35.039224 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:35.040110 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:35.052135 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:35.056012 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:35.057123 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:35.150965 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:35.151520 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:35.154052 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:35.160001 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:35.163295 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:35.164683 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:14:35.171821 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:35.182595 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:35.281678 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:35.288187 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:35.289065 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:35.289206 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:35.301064 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 14:14:35.306113 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:35.317098 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:35.319983 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:35.320710 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:35.324168 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:35.335018 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:35.335233 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:35.339075 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:35.340866 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:35.343073 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:35.368311 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:14:35.388437 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 14:14:35.389510 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 14:14:35.461001 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:35.462548 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:35.468912 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:35.479321 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:35.485682 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:35.488129 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:35.495873 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:35.527318 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:35.531121 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:35.531977 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:35.534162 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:35.534616 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:35.536989 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:35.556591 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:35.631969 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:35.633554 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:35.636507 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:35.638092 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:35.647589 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:35.652297 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:35.653755 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:35.707597 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:35.845579 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:35.845853 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:35.851524 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:35.854195 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:35.854301 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:35.855301 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:35.869650 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:35.979502 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:35.980382 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:35.985433 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:35.987226 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:35.991793 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:35.993142 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:35.994050 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:35.994529 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:36.002288 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:36.033357 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:36.035720 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:36.049306 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:36.054083 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:36.067448 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:36.136178 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:36.152724 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:36.153483 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:36.158399 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:36.158619 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:36.161506 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:36.169942 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:36.172441 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 14:14:36.178030 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:36.267311 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:36.270646 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:36.271906 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:36.272389 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:36.277304 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:36.280850 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:36.289780 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:36.377288 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:36.377835 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:36.380082 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:36.380280 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:36.382247 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:36.408694 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:36.424460 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:36.467482 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:36.469413 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:36.470252 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:36.470589 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:36.473232 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:36.479645 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:36.488903 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 14:14:36.507556 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:14:36.515358 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:36.548418 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:36.552955 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:36.557288 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:36.563997 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:36.564163 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:36.584986 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:36.586254 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:36.638938 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:36.640161 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:36.641477 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240501 14:14:36.645326 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:36.650195 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:36.652803 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 14:14:36.658844 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:36.665817 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:36.679697 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:36.813830 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:14:36.862516 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 14:14:36.883698 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:36.884536 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:36.885255 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:36.885382 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:36.886194 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:36.891355 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:36.911514 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:36.943624 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:14:36.954666 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 14:14:37.007790 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:37.008301 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:37.013850 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:37.019155 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:37.020989 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:37.026893 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:37.039484 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:37.049639 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:37.059232 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:37.063717 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:37.068938 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:37.070906 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:37.086103 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:37.102702 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:37.140376 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:14:37.165486 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 14:14:37.171661 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:37.173620 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:37.175803 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:37.177201 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:37.181008 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:37.183583 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:37.213533 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:37.286393 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:37.291193 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:37.291975 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:37.292734 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:37.294643 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:37.300530 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:37.313408 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:37.376034 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 14:14:37.407081 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:37.412292 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:37.418613 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:37.419216 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:14:37.431087 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 14:14:37.438732 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:37.443496 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:37.446978 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:37.449656 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:37.450246 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:37.450438 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:37.452265 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:37.461058 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:37.480810 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 14:14:37.488222 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:37.501634 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:37.549523 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:37.560113 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:37.566133 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:37.571979 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:37.572186 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:37.575636 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:37.578109 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:37.578404 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:37.627447 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:37.635645 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:37.639220 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:14:37.640748 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 14:14:37.641968 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:37.645900 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:37.647751 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:37.668069 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:37.894416 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:37.895196 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:37.896193 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:37.908107 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:37.912072 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:37.917673 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 14:14:37.922231 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:37.943491 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:37.945523 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:38.002538 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:38.005985 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:38.010679 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:38.013476 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:38.017153 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:38.021863 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:38.023118 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:38.072789 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 14:14:38.100329 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:38.100426 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:38.104739 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:38.106252 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:38.108105 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:38.127542 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:38.131743 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:38.177654 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:38.179464 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:38.180760 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:38.189261 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:38.189285 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:38.189533 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:38.202211 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:38.259675 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:38.263044 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:38.270246 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:38.271143 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:38.271965 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:38.273017 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:38.295133 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:38.329229 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:14:38.371428 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:38.371771 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:38.379500 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:38.381902 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:38.399166 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:38.400364 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:38.401001 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:38.445525 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:14:38.446205 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:38.484954 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:38.489923 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:38.495245 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:38.498353 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:14:38.498720 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 14:14:38.503192 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:38.504449 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:38.506094 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:38.558336 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:38.559741 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:38.561695 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:38.566031 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:38.568590 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:38.584014 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:38.591967 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:38.659811 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:38.660012 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:38.663785 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:38.668874 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:38.673995 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:38.681288 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:38.684486 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:38.791692 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 14:14:38.797945 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:38.802553 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:38.804505 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:38.807704 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:38.822003 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:38.831022 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:38.833817 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:38.887904 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:38.911672 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:38.915321 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:38.919952 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:38.922631 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:38.927003 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:38.932710 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:38.933737 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:38.983748 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 14:14:39.029364 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:39.030658 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:14:39.032218 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:39.039697 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:39.040859 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:39.047322 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:39.049213 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:39.053545 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:39.066493 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:39.074294 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:39.079188 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:39.080276 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:39.083629 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:39.089219 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:39.110651 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:39.115087 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:14:39.115793 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:14:39.148130 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 14:14:39.210786 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:39.214432 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:39.215206 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:39.221431 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:39.232271 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:39.246068 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:39.250365 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:39.258105 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:39.260918 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:39.264283 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:39.273154 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:39.283464 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:39.285655 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:39.307729 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:39.387497 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:39.389133 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:39.389739 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:39.393863 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:39.396212 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:39.420118 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:39.420344 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:39.443842 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:14:39.521193 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:14:39.585063 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 14:14:39.586736 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:39.589118 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:39.589831 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:39.595103 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:39.596062 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:39.597198 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:39.609819 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 14:14:39.627866 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:39.718997 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:39.727265 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:39.730013 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:39.731472 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:39.731931 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:39.734822 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:39.741737 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:39.756386 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:39.764151 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:39.776829 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:39.783210 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:39.783987 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 14:14:39.788221 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:39.789795 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:39.803014 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:39.891129 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:39.893600 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:39.902442 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:39.908963 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:39.908998 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:39.922170 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:39.931923 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:39.960783 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:14:39.985571 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 14:14:40.008553 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:40.010950 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:40.011312 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:40.018061 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:40.018821 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:40.028312 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:40.048796 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:40.126242 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:14:40.130403 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 14:14:40.135135 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:40.138701 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:40.142913 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:40.144472 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:40.150406 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:40.156790 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 14:14:40.159256 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:40.206025 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:40.206243 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:40.211596 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:40.219822 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:40.227356 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:40.235148 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:40.235574 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:40.294400 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:40.297111 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:40.301560 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:40.307317 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:40.308427 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:40.310145 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:40.319442 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:40.374162 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:40.374507 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:40.381322 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:40.385267 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:40.388144 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:40.391235 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:40.416002 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:40.475988 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 14:14:40.497845 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:40.607883 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:40.611205 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:40.613617 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:40.620805 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:40.623289 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:40.632464 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:40.645211 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:40.707808 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 14:14:40.729729 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:40.735185 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:40.746027 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:40.750102 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:40.750211 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:40.759234 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:40.760919 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:40.768124 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:40.778532 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:40.780192 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:40.791289 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:40.807329 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:40.816772 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:40.823510 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:40.841063 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:14:40.905773 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:40.909922 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:40.918666 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:40.921046 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:40.923590 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:40.924854 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:40.930639 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:40.937937 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:41.023463 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:41.026235 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:41.028116 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:41.036842 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:41.041576 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:41.042418 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:41.055707 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:41.138059 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:41.141631 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:41.144094 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:41.149649 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:41.153239 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:41.170058 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:41.170372 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:41.181816 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 14:14:41.215359 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:41.221341 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:41.222743 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:41.225543 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:41.226001 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:41.227319 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:41.239358 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:41.303926 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:41.307188 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:41.311851 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:41.317708 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:41.320237 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:41.320395 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:41.338227 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:41.388726 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:41.389041 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:41.392621 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:41.393757 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:41.399547 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:41.403307 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:41.409276 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:41.467203 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:14:41.498124 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:14:41.508335 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:14:41.594542 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:14:41.609848 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 14:14:41.632018 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:41.634959 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:41.635947 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:41.637606 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:41.638506 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:41.639477 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:41.660771 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:14:41.672024 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 18 similar messages]
I20240501 14:14:41.749378 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:41.751905 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:41.760875 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:41.763775 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:41.770735 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:41.771088 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:41.771494 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:41.797263 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:41.805859 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:41.811671 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:41.815186 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:41.815512 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:41.818548 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:41.828806 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:41.917846 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:41.921149 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:41.921343 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:41.922536 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:41.923715 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:41.927079 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:41.940829 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:41.959017 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 14:14:42.024134 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:42.025830 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:42.032703 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:42.033535 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:42.042532 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:42.042797 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:42.044570 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:42.131249 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:42.149140 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:42.152529 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:42.156595 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:42.162756 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:42.163657 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:42.175547 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:42.184890 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:42.200150 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:42.212970 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:42.215199 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:42.217288 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:42.220794 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:42.223686 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:42.246770 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:42.294121 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 14:14:42.305008 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:42.305500 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:42.306355 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:42.315335 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:14:42.317173 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 14:14:42.319389 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:42.324313 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:42.338958 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:42.347671 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:42.374684 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:42.377349 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:42.380889 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:42.389470 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:42.390769 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:42.404388 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:42.405398 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:14:42.491595 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:14:42.501809 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:14:42.621433 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:42.636364 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:42.638620 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:42.639657 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:42.646286 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:42.647136 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:42.649971 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:42.660094 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:42.741230 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 14:14:42.747721 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:42.751165 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:42.752519 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:42.757962 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:42.759496 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:42.767952 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:42.770953 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:42.774585 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:42.779702 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:42.787233 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:42.797823 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:42.805878 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:42.825569 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:42.825866 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:42.909569 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:42.914366 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:42.916002 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:42.923401 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:42.931434 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:42.934499 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:42.942715 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:43.003247 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 14:14:43.018349 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:43.019914 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:43.021509 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:43.026206 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:43.029166 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:43.032841 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:43.035918 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:43.084784 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:14:43.102895 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 14:14:43.136296 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:43.137259 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:43.142257 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:43.150969 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:43.152312 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:43.171222 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:43.176976 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:43.201674 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:43.202987 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:43.203291 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:43.207574 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:43.215400 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:43.226841 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:43.227746 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:43.314627 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:43.327349 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:43.329107 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:43.332475 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:43.333606 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:43.335322 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:43.363226 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:43.366083 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:43.373185 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:43.374557 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:43.385960 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:43.391319 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:43.391659 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:43.399431 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:43.542682 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:14:43.585007 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:14:43.647554 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:14:43.648248 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 14:14:43.676980 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:43.680444 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:43.682826 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:43.683190 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:43.686077 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:43.705629 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:43.713220 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 14:14:43.717005 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:43.758188 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:43.760105 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:43.767102 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:43.783706 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:43.785421 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:43.786013 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:43.795569 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:43.837566 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:43.847004 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:43.848984 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:43.853181 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:43.858678 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:43.877063 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:43.877511 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:43.943717 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:43.945201 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:43.950325 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:43.950964 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:43.952795 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:43.954044 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:43.979337 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:44.051031 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 14:14:44.055131 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:44.056628 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:44.057804 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:44.060906 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:44.065917 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:14:44.067732 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:44.085916 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:44.092954 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:44.188535 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:44.188756 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:44.189498 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:44.191234 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:44.194581 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:44.196807 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:44.208475 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 14:14:44.211474 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:44.222347 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:44.229606 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:44.231364 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:44.233704 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:44.239675 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:44.253764 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:44.273803 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:44.353974 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:44.358692 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:44.360270 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:44.360738 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:44.370602 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:44.372239 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:44.376931 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:44.380550 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 14:14:44.387199 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:44.411216 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:44.411731 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:44.413239 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:44.414851 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:44.420162 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:44.434757 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:44.524574 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:14:44.551820 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:14:44.552111 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:14:44.643600 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:14:44.655817 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:44.743084 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:44.744289 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:44.747228 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:44.747228 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:44.755393 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:44.756968 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:44.768057 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:44.773093 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:44.783115 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:44.784273 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:44.791886 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:44.798517 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:44.799122 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:44.817844 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:44.867200 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:14:44.896211 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 14:14:44.904268 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:44.906764 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:44.907379 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:44.916769 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:44.917989 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:44.925556 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 14:14:44.953023 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:44.956437 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:44.957072 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:44.963444 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:44.968379 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:44.971493 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:44.982333 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:44.987704 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:14:44.989030 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 14:14:44.999681 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:45.087400 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:45.089699 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:45.093941 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:45.094971 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:45.097868 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:45.098587 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:45.109987 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:45.162900 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 14:14:45.195727 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:45.199438 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:45.201831 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:45.205210 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:45.207577 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:45.207584 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:45.216472 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:45.219470 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:14:45.238333 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 14:14:45.313324 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:45.319110 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:45.330083 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:45.331801 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:45.337236 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:45.343398 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:45.351982 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 14:14:45.355530 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:45.363989 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:45.376924 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:45.377431 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:45.377704 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:45.381073 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:45.381932 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:45.408558 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:14:45.476377 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240501 14:14:45.480638 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:45.481230 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:45.483616 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:45.489701 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:45.489804 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:45.511338 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:45.513922 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:45.694550 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:45.697108 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:45.698434 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:45.707779 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:45.723963 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:45.725478 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:45.734690 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:45.765514 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:45.824026 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:45.824826 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:45.832353 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:45.832681 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:45.839357 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:45.841698 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:45.866827 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:45.878667 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:45.880053 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:45.891515 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:45.894228 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:45.899830 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:45.902150 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:45.906201 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 14:14:45.923924 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:45.989822 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:45.992254 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:46.003588 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:46.013005 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:46.016414 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:46.040894 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:46.043766 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:46.047643 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:46.052023 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:46.071676 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:46.079394 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:46.079998 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:46.085292 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:46.087456 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:46.179790 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:46.180595 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:46.182266 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:46.182909 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:46.190115 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:46.194715 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:46.201575 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:46.214211 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:14:46.241721 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 14:14:46.287457 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:46.287743 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:46.292722 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:46.297109 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:46.297868 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:46.298401 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:46.315637 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:46.382637 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:46.383554 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:46.385113 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:46.386301 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:46.394740 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:46.395308 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:46.408716 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:46.495680 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:46.499307 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:46.501955 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:46.510795 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:46.511543 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:46.516009 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:46.523962 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 14:14:46.534989 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:46.546201 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 14:14:46.604271 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:46.607192 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:46.611189 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:46.612110 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:46.612522 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:46.613916 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:46.624594 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:46.634323 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:46.726900 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:46.729246 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:46.734026 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:46.736358 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:46.737212 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:46.739728 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:14:46.768702 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240501 14:14:46.770413 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:46.854197 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:46.854454 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:46.857470 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 14:14:46.859537 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:46.865834 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:46.870039 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:46.876116 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:46.891192 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:46.919963 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:46.921018 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:46.926203 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:46.935185 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:46.935612 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:46.954365 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:46.957731 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:46.958457 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 14:14:47.044950 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:47.055446 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:47.058013 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:47.063179 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:47.077517 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:47.086861 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:47.091318 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 14:14:47.091763 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:47.096109 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:47.102128 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:47.107182 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:47.107277 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:47.121917 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:47.123507 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:47.129176 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:47.202891 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 14:14:47.210783 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:47.216220 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:47.216691 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:47.217273 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:47.218127 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:47.220435 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:47.222395 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:47.229102 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:47.282892 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:14:47.404621 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 14:14:47.425437 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:47.425436 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:47.428490 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:47.435173 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:47.439775 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:47.440440 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:47.467729 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:47.540535 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:47.545099 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:47.548840 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:47.554034 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:47.561419 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:47.563257 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 14:14:47.565750 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:47.575532 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:47.603554 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:47.609467 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:47.612244 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:47.615329 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:47.617825 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:47.629185 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:47.635186 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:47.718760 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:47.722474 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:47.726186 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:47.727111 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:47.727305 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:47.728190 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:47.746568 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:14:47.756229 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:14:47.768477 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:14:47.800266 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:14:47.808332 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:47.830962 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:47.832630 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:47.835847 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:47.836311 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:47.847906 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:47.852381 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:47.864202 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:47.964066 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:47.966558 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:47.969385 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:47.970963 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:47.971004 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:47.981782 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:48.005970 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:48.010895 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:48.022831 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:48.023456 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:48.026358 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:48.030239 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:48.032817 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:48.033231 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:48.140909 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:48.148995 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:48.154877 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:48.156029 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:48.169871 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:48.170903 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:48.171873 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:48.172214 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:48.174419 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:48.194761 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:48.198319 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:48.203339 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:48.205497 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:48.206172 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:48.224264 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:48.520695 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:48.522563 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:48.527747 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:48.531950 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:48.537977 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:48.539626 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:48.540026 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:48.543200 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:48.551218 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:48.570942 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:48.576936 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:48.577222 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:48.587276 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:48.599277 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:48.634205 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:14:48.641150 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:14:48.684990 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 14:14:48.695269 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:48.697428 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:48.701072 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:48.703898 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:48.709647 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:48.724798 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:48.737332 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:48.743448 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:48.745136 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:48.745311 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:48.748337 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:48.758729 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:48.766196 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:48.775660 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:48.778854 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 14:14:48.779588 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:14:48.860025 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 14:14:48.865664 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:48.866017 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:48.866334 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:48.875202 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:48.882202 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:48.890053 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 14:14:48.893180 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:48.899942 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:48.984786 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:48.986951 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:48.990356 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:48.993641 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:48.994091 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:49.001070 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:49.009063 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:49.092090 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:49.094184 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:49.094969 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:49.099931 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:49.101222 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:49.103601 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:49.114621 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:49.168692 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:49.170740 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:49.181413 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:49.181847 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:49.185668 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:49.185667 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:49.192315 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:49.249395 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:49.251909 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:49.256938 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:49.258399 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:49.259598 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:49.263345 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:49.279116 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:49.281308 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:49.293742 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 14:14:49.489109 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:49.492236 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:49.493888 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:14:49.497372 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 14:14:49.501534 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:49.508117 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:49.510073 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:49.516893 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:49.592414 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:49.595746 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:49.599897 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:49.603125 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:49.603477 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:49.636770 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:49.647145 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:49.649348 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:49.656034 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:49.666988 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:49.669351 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:49.682380 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:49.683012 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:49.687902 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:49.726428 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 14:14:49.772802 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:49.778827 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:49.781908 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:49.783810 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:49.785214 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:49.785909 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:49.788466 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:14:49.789784 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:49.813226 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:14:49.838120 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 14:14:49.849908 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:49.854023 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:49.854966 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:49.855240 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:49.867144 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:49.871210 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:49.874763 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:49.918949 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 14:14:49.974850 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:49.976238 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 14:14:49.977592 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:49.981180 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:49.981763 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:14:49.987262 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:49.987486 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:49.999608 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:50.004369 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:50.103297 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:50.105069 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:50.108989 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:50.109488 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:50.110194 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:50.123601 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:50.132691 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:14:50.141388 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 14:14:50.157466 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:50.159415 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:50.171772 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:50.175001 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:50.178264 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:50.189766 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:50.190541 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:50.225021 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:50.274998 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:50.279467 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:50.279560 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:50.281836 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 14:14:50.286669 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:50.291570 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:50.291596 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 14:14:50.295334 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:50.304025 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:50.436568 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:50.438661 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:50.442095 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:50.445255 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:50.450683 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:50.461871 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:50.463294 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:50.550733 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:50.554360 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:50.560914 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:50.561043 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:50.561039 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:50.583999 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 14:14:50.584780 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:50.588322 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:50.638711 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:50.641670 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:50.647397 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240501 14:14:50.648847 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:50.650929 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:50.651053 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:50.653776 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:50.675981 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:50.725744 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:50.729363 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:50.729992 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:50.735811 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:50.737756 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:50.761395 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:50.765734 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:50.782090 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:50.808928 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:50.809459 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:50.810655 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:50.811558 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:50.811661 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:50.818886 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:50.847347 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:50.893011 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:50.893278 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:14:50.901602 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 14:14:50.903201 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:50.905215 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:50.909329 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:50.924386 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:50.925665 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:51.001947 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:51.002148 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:51.008332 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:51.013186 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:51.015877 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:51.015967 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:51.021112 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:51.056447 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 14:14:51.131279 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:51.135186 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:51.139624 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:51.140275 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:51.143347 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:51.145764 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:51.164616 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:51.176357 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:14:51.177597 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 14:14:51.243649 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:51.244323 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:51.248060 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:51.248734 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:51.256446 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:51.259233 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:51.264817 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:51.330153 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:51.353940 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:51.357079 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:51.366950 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:51.368144 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:51.368211 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:51.369266 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:51.370308 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:51.469471 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:51.474293 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:51.474570 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:51.481284 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:51.484843 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:51.487794 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:51.490556 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:51.591007 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:51.591413 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:51.595968 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:51.603660 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:51.615294 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:51.622363 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:51.630127 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:51.631697 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:51.634173 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:51.635082 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:51.638484 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:51.647253 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:51.650882 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:51.662281 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:51.749960 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:51.752740 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:51.756556 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:51.757421 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:51.770641 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:51.773182 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:51.802790 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:51.806149 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:51.816220 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:51.819566 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:51.824321 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:51.826013 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:51.826565 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:51.841250 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:51.858738 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:14:51.918460 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 14:14:51.921283 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:51.922554 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:51.923611 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:51.928843 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:51.929883 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:51.932590 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:51.940833 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:51.941592 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:14:52.055874 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:14:52.076265 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 14:14:52.127228 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:52.127615 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:52.132484 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:52.135380 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:52.140373 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:52.153908 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:52.165365 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:52.202387 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 14:14:52.243635 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:52.247654 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:52.253310 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:52.254122 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:52.259374 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:52.281966 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:52.283164 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:52.292770 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:52.306756 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:52.307201 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:52.308892 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:52.310622 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:52.319610 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:52.324204 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:52.326102 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:52.423172 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:52.423274 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:52.424049 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:52.424543 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:52.427727 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:52.440346 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 14:14:52.448311 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:52.458716 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:52.533502 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:52.534991 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:52.535199 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:52.542407 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 14:14:52.544581 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:52.545017 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:52.552589 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:52.556851 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:52.653666 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:52.655730 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:52.667454 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:52.670542 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:52.675379 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:52.685458 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:52.686094 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:52.691087 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:52.706517 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:52.707311 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:52.707644 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:52.708905 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:52.721530 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:52.732134 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:52.773352 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:14:52.791766 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 14:14:52.835629 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:52.843761 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:52.843806 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:52.845242 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:14:52.847664 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 14:14:52.855563 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:52.858300 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:52.858345 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:52.870059 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:52.885823 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:52.890076 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:52.891499 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:52.906641 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:52.916430 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:52.917591 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:52.919181 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:53.045224 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:14:53.160421 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 14:14:53.210425 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:53.214114 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:53.215492 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:53.225901 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:53.227519 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:53.229125 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:53.232767 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:53.277456 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:53.277793 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:53.280954 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:53.281211 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:53.290256 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:53.290318 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:53.320719 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:53.391196 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:53.392871 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:53.399509 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:53.405336 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:53.407627 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:53.413563 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:53.414842 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:53.417312 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:53.476392 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:53.481230 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:53.484158 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:53.484678 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:53.485013 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:53.500432 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:53.520515 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:53.596446 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 14:14:53.596729 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:53.601594 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:53.604365 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:53.607002 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:53.607214 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:53.631410 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:53.631812 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:53.693317 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:14:53.700541 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:14:53.709208 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240501 14:14:53.734874 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:53.748842 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:53.750698 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:53.751421 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:53.752812 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:53.760375 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:53.761293 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:53.790606 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:53.794489 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:53.797385 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:53.798420 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:53.821024 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:53.830091 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:53.830719 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:53.905717 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:53.908538 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:53.908882 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:53.911135 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:53.913476 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:53.915026 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:53.923516 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:53.926025 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:53.974234 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:53.978231 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:53.983261 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:53.986331 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:53.986570 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:53.989931 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:54.006590 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:54.086032 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:14:54.228297 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:54.234973 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:54.236199 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:54.241539 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:54.241608 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:54.242158 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:54.249733 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:54.310695 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:14:54.324452 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:14:54.336144 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 14:14:54.351119 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:54.351836 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:54.356514 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:54.356932 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:54.376030 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 14:14:54.380306 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:54.387823 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:54.392820 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:54.398787 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:54.410109 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:54.410794 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:54.421291 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:54.428952 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:54.429369 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:54.442446 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:54.521374 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:54.522320 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:54.523831 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:54.528707 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:54.543860 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:54.552790 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:54.576049 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:54.576252 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:54.580788 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:54.583556 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:54.592325 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:54.596292 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:54.601336 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:54.624708 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:54.716423 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:54.724344 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:54.726647 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:54.727208 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:54.735857 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:54.745203 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:54.749987 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:54.787878 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:14:54.817955 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:54.834949 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:54.838351 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:54.838349 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:54.840078 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:54.841670 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:54.860481 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:54.874799 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:54.897768 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 14:14:54.944001 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:54.947752 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:54.948441 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:54.956537 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:54.961405 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:54.969714 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:54.974081 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:55.036762 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 14:14:55.062733 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:55.063120 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:55.065979 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:55.069505 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:55.071066 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:14:55.084553 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:14:55.091326 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:55.095183 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:55.105978 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 14:14:55.171010 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:55.174412 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:55.175338 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:55.177527 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:55.188114 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:55.191197 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:55.192857 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:55.234012 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:14:55.258355 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 14:14:55.285876 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:55.288342 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:55.294459 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:55.297147 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:55.305153 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:14:55.305166 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:55.305827 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 14:14:55.306174 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:55.306418 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:55.403419 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:55.411015 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:55.422936 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:55.425462 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:55.426831 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:55.436005 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:55.440786 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:55.450953 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:55.471338 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:55.483016 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:55.487166 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:55.490115 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:55.490502 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:55.497359 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:55.506728 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 14:14:55.584172 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:55.590751 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:55.593899 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:55.595207 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:55.596623 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:55.601219 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:55.621558 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:55.634025 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:55.648676 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:55.650986 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:55.656492 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:55.661285 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:55.667286 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240501 14:14:55.676436 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:55.676800 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:55.687695 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:55.749401 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 14:14:55.766140 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:55.769665 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:55.771957 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:55.772714 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:55.782884 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:55.783488 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:55.797358 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:55.840881 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 14:14:55.941608 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:55.943506 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:55.947292 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:55.948482 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:55.957897 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:55.959379 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:55.960170 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:55.971305 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 14:14:56.052771 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:56.054824 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:56.076638 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:56.081338 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:56.084671 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:56.084676 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:56.085515 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:56.085587 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:56.145615 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:56.149660 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:56.150455 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:56.151399 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:56.177618 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:56.178395 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:56.178622 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:56.242723 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 14:14:56.264470 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:56.270284 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:56.275482 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:56.276347 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:56.278499 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:56.292865 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:56.292953 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:56.391805 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:56.393527 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:56.393925 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:56.404239 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:56.417889 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:56.421798 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:56.431517 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:56.436758 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:56.456426 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:56.458155 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:56.470670 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:56.473886 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:56.474161 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:56.488816 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:56.563824 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:56.565922 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:56.567325 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:56.577409 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:56.577488 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:56.581682 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:56.592468 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:56.598066 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:56.638574 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:56.639812 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:56.641435 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:56.648021 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:56.653748 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:56.655782 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:56.658142 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:56.716682 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:56.719282 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:56.719669 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:56.723711 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:56.728066 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:56.733515 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:56.752979 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:56.837540 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 14:14:56.959023 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:56.960032 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:56.967932 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:56.969864 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:56.971613 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:56.972054 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:56.998574 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:57.052604 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 14:14:57.088248 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:57.090484 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:57.095435 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:57.103227 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:57.105754 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:57.108243 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:57.126907 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:57.131719 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:57.133586 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:57.135197 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:14:57.145480 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 14:14:57.151633 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:14:57.152887 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 14:14:57.153688 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:57.160468 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:57.173326 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:57.191689 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:57.265784 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:57.269611 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:57.272101 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:57.273720 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:57.274453 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:57.279801 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:57.299269 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:57.375169 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:57.376138 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:57.376247 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:57.378077 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:57.390493 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:57.398025 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:57.411178 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:57.444761 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:14:57.461047 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 14:14:57.487344 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:57.491223 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:57.495200 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:57.498890 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:57.503304 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:57.503566 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:57.519853 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:14:57.541098 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 14:14:57.569283 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:57.572522 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:57.576159 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:57.582194 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:57.584724 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:57.585096 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:57.603309 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:57.618880 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 14:14:57.669649 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:57.671192 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:57.671379 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:57.673935 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:57.683650 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:57.685710 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:57.694403 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:57.704591 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:57.741621 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:57.745381 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:57.751765 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:57.753413 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:57.760076 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:57.762701 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:57.777395 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:57.800860 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:14:57.907568 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:14:57.986042 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 14:14:58.011761 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:58.013198 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:58.013404 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:58.013872 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:58.027899 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:58.039520 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:58.041149 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:58.116204 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:58.117014 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:58.120132 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:58.127431 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:58.133188 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:58.133298 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:58.146714 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:58.178838 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:58.182119 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:58.190310 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:58.195017 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:58.199451 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:58.204625 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:58.206166 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:58.239352 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:58.291749 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:58.293128 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:58.293910 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:58.294711 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:58.296449 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:58.306785 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:58.317112 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 14:14:58.327479 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:58.340212 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 14:14:58.394008 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:58.394431 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:58.404281 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:58.407765 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:58.409619 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:58.410991 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:58.441540 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:58.516510 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:58.516588 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:58.521567 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:58.529814 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:58.535235 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:58.540802 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:58.551752 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:58.571723 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:58.582165 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:58.585454 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:58.587395 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:58.589038 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:58.599294 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:58.610558 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:58.635529 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 14:14:58.679773 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:58.682678 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:58.698945 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:58.701026 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:58.703042 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:58.714233 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:14:58.716862 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240501 14:14:58.742200 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:58.744817 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:58.749652 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:58.750355 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:58.753002 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:58.758683 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:58.768069 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:58.787765 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:58.839219 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:58.846170 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:59.089622 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:59.092056 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:59.095839 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:59.096125 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:59.098385 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:59.101027 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:59.119593 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:59.121093 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:59.126010 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:59.136263 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:59.138686 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:59.139662 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:59.160033 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:59.168684 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:14:59.187937 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 14:14:59.244355 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 14:14:59.263548 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:59.273779 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:59.274369 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:59.274389 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:59.289700 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:59.302560 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:59.311316 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:59.311748 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:59.313611 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:59.317260 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:59.324798 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:59.327592 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:59.340130 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:59.341733 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:59.365679 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:59.385002 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:14:59.419800 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 14:14:59.430579 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:59.431783 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:59.435534 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:59.435637 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:59.450664 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:59.451125 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:59.455421 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:59.535681 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:59.536504 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:59.536652 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:59.538916 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:59.539217 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:59.550951 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:59.561553 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:59.633294 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:59.640210 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:59.643411 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:59.643750 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:59.644340 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:59.653211 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:59.669931 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:59.702508 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:14:59.716317 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 14:14:59.717947 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:59.723870 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:59.726943 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:59.727573 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:59.728554 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:59.744386 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:59.778951 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:14:59.796573 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:14:59.796975 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:14:59.803472 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:59.806826 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:14:59.807245 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:14:59.810798 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:14:59.811825 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:14:59.819566 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 14:15:00.029314 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:00.030257 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:00.032312 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:00.037427 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:00.043330 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:00.043956 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:00.055356 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 14:15:00.059044 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:00.143904 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:00.146116 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:00.148939 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:15:00.153671 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 14:15:00.154124 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:00.154646 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:15:00.158331 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 14:15:00.160699 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:00.161008 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:00.211903 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:00.213678 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:00.218027 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:00.219130 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:00.224076 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:00.239913 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:15:00.256623 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 14:15:00.256759 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:15:00.289103 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:15:00.301796 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:00.302402 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:00.305542 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:00.305966 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:00.312649 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:00.312852 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:00.315738 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:00.316385 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:00.378626 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:00.381106 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:00.385800 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:00.389338 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:00.390203 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:00.407392 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:00.416236 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:00.503098 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:00.505728 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:00.508749 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:00.510200 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:00.512308 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:00.524829 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:00.530768 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:15:00.552091 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 14:15:00.615602 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:00.621552 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:00.624192 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:00.632947 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:00.635656 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:00.643221 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:00.647179 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:00.664403 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:00.673890 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:00.682505 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:00.683281 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:00.686703 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:00.690418 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:00.695086 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:15:00.701431 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:15:00.753912 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240501 14:15:00.785275 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:00.785580 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:00.786686 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:00.789861 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:00.792536 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:00.794106 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:00.821698 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:00.858489 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:15:00.924077 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:00.985199 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:00.985487 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:00.987676 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:00.989647 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:01.010165 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:01.010958 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:01.014092 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:15:01.040716 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 14:15:01.117245 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:01.120805 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:01.124869 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:01.132683 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:01.134883 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:01.136296 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:01.142865 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:01.147580 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:01.147693 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:01.152103 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:01.158286 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:01.170894 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:01.175710 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:01.191327 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:15:01.240041 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:15:01.257804 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 14:15:01.298676 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:01.299909 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:01.309486 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:01.313014 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:01.313802 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:01.319940 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:01.324863 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:01.329414 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:01.333169 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:01.334604 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:01.334712 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:01.360540 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:01.364506 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:01.374269 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:15:01.430634 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:01.462713 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:01.464380 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:01.468818 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:01.479902 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:01.486896 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:01.487965 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:01.497079 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:01.558202 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 14:15:01.572785 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:01.572978 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:01.578419 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:01.581615 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:01.588945 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:01.600433 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:01.604034 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:01.678772 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:01.679054 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:01.681211 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:01.687602 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:01.690569 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:01.705607 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:01.708518 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:01.781358 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:01.781900 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:01.785811 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:01.788204 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:01.798589 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:01.798920 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:01.822443 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 14:15:01.825075 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:15:01.848431 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:01.899794 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:01.902065 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:01.905902 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:01.910151 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:01.916774 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:01.929067 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:01.939155 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:02.017653 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:02.019582 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:02.023015 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:02.023645 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:02.027756 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:02.029594 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:15:02.041158 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 14:15:02.062690 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:02.092103 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 14:15:02.138471 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:02.141458 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:02.141657 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:02.142349 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:02.158953 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:02.165578 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:02.175833 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:02.177163 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:02.199406 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:02.201514 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:02.203548 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:02.203895 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:02.208585 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:02.210795 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:02.300647 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:02.301241 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:02.306602 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:02.307993 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:15:02.308895 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 14:15:02.321734 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:02.327706 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:02.341809 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:02.360540 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:02.362394 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:02.367112 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:02.367369 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:02.376974 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:02.390827 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 14:15:02.392941 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:02.397009 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:02.401575 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:15:02.468652 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:02.472105 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:02.472127 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:02.474027 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:02.475643 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:15:02.478662 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240501 14:15:02.483558 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:02.493851 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:02.505831 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:15:02.576975 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 14:15:02.651863 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:02.652611 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:02.658074 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:02.659515 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:02.660063 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:02.663842 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:15:02.671489 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 14:15:02.677726 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:02.752138 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:02.755188 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:02.759459 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:02.762362 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:02.770320 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:02.775451 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:02.785049 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:02.845260 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:02.847865 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:02.848630 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:02.856566 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:02.856765 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:02.856787 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:02.857002 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:02.857295 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:15:02.899492 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 14:15:02.960038 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:02.960520 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:02.960747 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:02.961383 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:02.973501 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:02.980437 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:02.981312 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:03.082959 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:03.088485 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:03.088534 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:03.090546 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:03.101706 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:03.110142 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:03.111320 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:03.121251 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:03.126008 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:03.136503 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:03.140234 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:03.143133 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:03.161947 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:03.174723 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:03.250689 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:03.255179 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:15:03.256542 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 14:15:03.261329 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:03.266543 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:03.272387 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:03.283975 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:03.298607 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:03.311456 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:03.316017 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:03.319368 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:03.322398 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:03.323606 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:03.328599 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:03.366041 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 14:15:03.366387 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:03.403908 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:03.406572 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:03.412634 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:03.413938 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:15:03.424389 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:03.426115 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:03.428440 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:03.438716 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:03.649857 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:03.649910 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:03.651423 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:03.651908 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:03.653774 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:03.657109 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:03.665663 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:15:03.758958 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 14:15:03.760758 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:03.766350 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:03.767604 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:03.771415 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:03.773005 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:03.784073 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:03.787855 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:03.794975 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:03.801997 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:03.813486 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:03.820195 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:03.835897 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:03.837006 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:03.855731 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:15:03.864120 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 14:15:03.928017 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:03.929204 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:03.936780 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:03.943436 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:03.946139 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:03.946975 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:03.947707 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:15:04.021066 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:04.052193 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:04.061525 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:04.062681 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:04.063369 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:04.069278 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:04.087332 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:04.090742 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:15:04.129273 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 14:15:04.164997 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:04.166954 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:04.166954 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:04.178488 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:04.180670 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:04.185094 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:04.213863 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:04.237684 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:04.240367 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:04.247490 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:15:04.248363 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:15:04.251204 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:04.252303 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:04.263885 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:04.264297 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:04.313918 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 14:15:04.351972 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:04.354138 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:04.365787 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:04.367195 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:04.367727 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:04.369382 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:04.391988 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:04.394716 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:04.395040 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:04.396675 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:04.402997 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:15:04.403573 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 14:15:04.415956 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:04.417804 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:04.442628 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:15:04.543155 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:15:04.624418 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:15:04.688622 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 14:15:04.741091 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:04.741962 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:04.742717 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:04.757481 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:04.760622 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:04.761927 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:04.766289 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:04.766947 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:04.774400 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:04.774502 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:04.776996 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:04.782034 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:15:04.784261 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 14:15:04.807689 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:04.819669 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:15:04.837074 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:15:04.911590 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 14:15:04.929427 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:04.937420 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:04.939116 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:04.939616 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:04.948137 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:04.958374 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:04.959704 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:04.963964 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:04.975289 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:04.981653 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:04.981775 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:04.984468 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:04.984575 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:05.009477 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:05.053493 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:15:05.085253 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 14:15:05.102593 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:05.106866 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:05.109007 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:05.113838 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:05.114495 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:05.116391 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:05.117113 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:05.121384 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:05.222718 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:05.223482 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:05.226799 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:05.228861 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:05.230250 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:05.237612 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:05.250280 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:05.276582 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 14:15:05.342944 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:05.348464 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:05.354861 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:05.359135 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:05.362429 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:05.365046 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:05.365852 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:05.376461 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:05.385726 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:05.395082 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:05.404603 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:05.407182 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:05.419533 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:05.420859 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:05.519917 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:05.520280 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:05.529390 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:05.534193 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:05.534684 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:05.535395 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:05.541165 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:05.610158 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 14:15:05.611605 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:05.737766 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:05.740567 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:05.746842 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:05.748345 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:05.750391 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:05.751989 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:05.755369 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:15:05.818879 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240501 14:15:05.853837 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:05.855028 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:05.860527 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:05.865159 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:05.872475 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:05.880079 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:05.884428 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:15:05.917690 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 14:15:05.921782 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:05.926105 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:05.937602 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:05.938238 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:05.944309 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:05.952917 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:05.963864 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:05.986902 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 14:15:06.029131 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:06.031916 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:06.033347 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:06.039222 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:06.045223 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:06.046983 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:06.075575 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:15:06.103358 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 14:15:06.129261 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:06.132014 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:15:06.134593 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:06.135955 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:06.141688 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:06.142369 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:15:06.149871 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:15:06.156679 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:06.163473 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:06.249337 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:06.251526 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:06.253460 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:06.259951 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:06.261991 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:06.283579 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:06.286796 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:06.377451 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:06.380451 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:06.389412 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:06.390718 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:06.391775 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:06.394886 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:06.404047 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:06.417317 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:06.433396 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:06.437858 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:06.441553 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:06.447530 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:06.449144 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:06.455724 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:06.542790 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:06.542850 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:06.545652 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:06.547962 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:06.551240 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:06.561002 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:06.572170 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:15:06.584570 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:15:06.677939 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 14:15:06.761246 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:06.767398 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:06.767938 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:06.771673 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:06.774825 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:06.776782 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:06.779522 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:06.828780 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 14:15:06.890625 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:06.891659 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:06.892267 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:06.894748 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:15:06.909734 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 14:15:06.910524 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:06.920444 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:06.921895 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:06.938979 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 14:15:06.968669 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:06.968853 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:06.968967 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:06.976248 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:06.977918 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:06.985642 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:07.013528 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:07.061273 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:07.067104 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:07.073174 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:07.078341 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:07.084970 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:07.097201 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:07.103556 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:07.123916 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:07.152074 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:07.156034 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:07.158694 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:07.158962 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 14:15:07.159772 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:07.161195 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:07.168942 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:07.177640 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:07.186230 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 14:15:07.274262 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:07.275437 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:07.275599 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:07.276861 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:07.280710 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:07.286334 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:07.309705 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:07.390623 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:07.393102 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:07.393435 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:07.411181 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:07.414332 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:15:07.420627 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 14:15:07.431023 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:07.433066 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:07.458886 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:07.460294 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:07.460738 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:07.482309 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:07.483788 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:07.490198 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:15:07.492228 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 14:15:07.495306 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:15:07.520987 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:15:07.531443 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:15:07.554051 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 14:15:07.567309 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:07.568920 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:07.577114 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:07.579308 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:07.588106 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:07.593924 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:07.594048 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:07.650803 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 14:15:07.751825 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:07.752192 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:07.752732 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:07.752985 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:07.761646 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:07.763686 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:07.772352 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:07.868494 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:07.869488 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:07.874289 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:07.878279 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:07.887104 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:07.888226 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:07.910749 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:07.941120 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:07.944729 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:07.949923 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:07.952641 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:07.971155 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:15:07.972365 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 14:15:07.979336 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:07.982434 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:07.989418 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:08.041749 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:08.045924 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:08.045960 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:08.050405 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:08.061383 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:08.074004 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:08.079684 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:08.127245 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:08.133504 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:08.140700 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:08.141590 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:08.143003 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:08.147356 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:08.158113 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:08.221992 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:08.225668 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:08.226445 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:08.228096 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:08.228587 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:08.230082 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:08.233561 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:08.328577 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:08.330581 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:08.337446 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:08.338966 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:08.341894 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:08.351991 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:15:08.357998 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 14:15:08.361093 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:15:08.402040 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 14:15:08.450378 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:08.451627 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:08.453116 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:08.455773 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:15:08.458086 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:08.465991 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:08.466866 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:08.481205 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:08.550899 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:08.554852 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:08.559199 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:08.559489 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:08.560186 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:08.562819 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:08.577812 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 14:15:08.578342 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:08.657416 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:08.658325 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:08.661008 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:08.667752 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:08.668102 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:08.672356 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:08.687839 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:08.776551 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:08.777416 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:08.781709 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:08.788182 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:08.796319 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:08.797691 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:08.810091 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:08.910197 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:08.912227 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:08.916972 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:08.918568 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:08.921051 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:08.931803 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:08.943141 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:08.943336 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:08.949554 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:08.951293 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:08.957921 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:08.961505 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:08.985576 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:08.992040 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:15:09.021535 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:15:09.022796 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:09.072229 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:09.073494 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:09.079214 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:09.087337 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:09.088580 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:09.106463 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:09.116578 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:09.139449 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:09.148241 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:09.149148 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:09.149920 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:09.153414 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:09.157089 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:09.162045 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:09.251621 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:09.258105 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:09.258200 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:09.263190 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:09.265084 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:15:09.271968 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 14:15:09.277073 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:09.290695 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:09.293952 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:15:09.324090 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:15:09.351635 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 14:15:09.413282 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:09.420182 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:09.422549 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:09.430032 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:09.430605 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:09.431190 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:09.433710 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:15:09.473570 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:09.526067 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:09.526437 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:09.527119 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:09.537568 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:09.537778 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:09.538515 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:09.549373 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:15:09.556547 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 14:15:09.614482 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:09.616947 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:09.619199 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:09.621284 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:09.621953 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:09.622280 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:15:09.639626 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 14:15:09.648483 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:09.692453 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 14:15:09.730825 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:09.734828 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:09.740641 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:09.746292 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:09.746516 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:09.749442 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:09.763190 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:09.858831 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:09.861544 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:15:09.862836 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 14:15:09.864254 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:09.864614 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:09.865319 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:09.869235 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:09.878856 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:09.886849 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:09.891768 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:09.893488 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:09.899410 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:09.915514 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:09.935101 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:09.935669 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:09.937891 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:09.938117 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 14:15:10.032022 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:10.034940 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:10.036077 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:10.041219 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:10.047179 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:15:10.055902 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 14:15:10.055959 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:10.062649 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:10.071102 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:10.073192 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:10.078572 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:10.097008 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:10.102406 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:10.111083 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:10.133574 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:10.141652 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 14:15:10.206894 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:10.207617 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:10.211715 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:10.221997 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:10.222878 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:10.222882 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:10.231722 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:10.391826 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:10.397541 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:10.398308 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:10.398730 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:10.401741 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:10.402407 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:10.417321 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:10.441378 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:15:10.447726 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:10.524698 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:10.530530 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:10.530790 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:10.533143 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:10.534932 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:10.546990 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:10.548084 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:10.563143 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:10.567492 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:10.584985 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:10.588819 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:10.592198 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:10.593838 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:10.600479 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:10.696174 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:10.703577 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:10.703958 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:10.707726 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:10.716076 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:10.716327 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:10.720979 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:10.803555 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:10.807368 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:10.815161 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:10.819769 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:10.822860 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:10.826246 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:15:10.837580 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:10.855511 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:10.875157 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 14:15:10.875913 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:15:10.887022 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 14:15:10.921797 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:10.927240 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:10.932643 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:10.941452 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:10.942659 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:15:10.945061 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 14:15:10.948755 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:10.963300 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:10.981068 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:10.992775 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:10.996826 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:10.999078 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:11.002703 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:11.006742 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:11.025058 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:15:11.084935 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:15:11.099148 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:11.108345 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:11.108764 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:11.108801 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:11.114625 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:11.123471 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:11.124433 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:11.167898 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:11.170567 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:11.171519 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:11.174115 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:11.174157 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:11.176723 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:11.187161 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:11.291482 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 14:15:11.389060 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:11.426455 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:11.430449 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:11.433373 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:11.439249 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:11.443082 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:11.443202 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:11.475241 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:11.521661 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:11.524158 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:11.524220 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:11.529146 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:11.538412 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:11.551597 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:11.560058 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:11.607453 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:11.607627 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:11.609058 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:11.612013 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:11.634956 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:11.635243 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:11.636616 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:11.690342 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:11.692979 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:11.695523 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:11.707553 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:11.709997 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:11.710321 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:11.710357 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:15:11.756201 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 14:15:11.807072 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:11.808990 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:11.820590 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:11.822316 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:11.834131 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:11.834168 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:11.834553 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:11.943504 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:11.949646 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:11.957767 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:11.959656 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:11.961213 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:11.967051 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:11.970067 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:11.972841 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:11.975376 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:11.976471 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:11.990038 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:11.992689 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:12.000331 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:15:12.003875 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 14:15:12.012027 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:12.012516 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:15:12.018738 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 14:15:12.115686 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:12.119351 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:12.121011 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:12.128923 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:12.130970 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:12.139245 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:12.142939 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:12.156510 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:12.157598 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:12.169894 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:12.183790 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:12.183974 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:12.187208 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:12.195164 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:12.216112 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:15:12.220196 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:15:12.237771 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:15:12.385514 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:15:12.395778 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:15:12.485471 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:12.507645 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:12.508735 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:12.511344 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:12.519923 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:12.524185 21492 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:12.524405 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:12.526899 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:12.527235 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:12.536994 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:12.538507 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:12.543396 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:15:12.551220 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 14:15:12.564240 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:12.570794 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:12.573014 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:15:12.624109 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 14:15:12.678112 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:12.680580 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:12.682775 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:12.686518 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:12.691987 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:12.705519 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:12.714339 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:12.734089 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:15:12.738914 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 14:15:12.748148 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:12.749879 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:12.752807 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:12.753439 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:12.756076 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:15:12.771144 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 14:15:12.774153 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:12.846184 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:12.849176 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:12.856580 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:12.857578 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:12.858853 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:12.867178 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:15:12.875741 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:12.876451 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:12.953327 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:12.956565 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:12.956843 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:12.961110 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:12.961314 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:12.974366 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:12.988068 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:13.071997 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:13.074398 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:13.082263 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:13.088536 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:13.093168 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:13.093416 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:13.110493 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:13.144320 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:13.152127 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:13.153153 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:13.155519 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:13.156836 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:13.156976 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:13.160161 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:15:13.184813 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 14:15:13.236205 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:13.238893 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:13.239638 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:13.240705 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:13.242805 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:13.246845 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:13.250100 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:15:13.373750 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:15:13.390096 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 14:15:13.466940 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:15:13.466984 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 14:15:13.472903 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:13.473691 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:13.475935 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:13.484899 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:13.493072 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:13.493656 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:13.574491 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:13.574692 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:13.583267 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:13.584954 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:13.588253 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:13.593597 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:13.598474 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:13.673837 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:13.674156 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:13.681649 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:13.684146 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:13.685277 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:13.703614 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:13.707206 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:13.777603 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:13.779593 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:13.784174 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:13.792029 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:13.793362 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:13.797868 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:13.808043 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:13.853185 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:13.853765 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:13.856113 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:13.856118 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:13.861980 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:13.876139 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:13.885543 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:13.896620 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:13.967573 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:13.969179 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:13.972515 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:13.979568 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:13.980506 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:13.993775 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:13.997932 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:15:14.058020 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 14:15:14.082000 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:14.084357 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:14.088475 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:14.088822 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:14.090269 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:14.109347 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:14.121721 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:14.161499 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:14.163306 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:14.164677 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:14.169885 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:14.171119 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:14.173755 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:14.180855 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240501 14:15:14.187116 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:14.278110 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:14.278297 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:14.287722 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:14.288453 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:14.288533 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:14.290232 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:14.303519 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:14.389411 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:15:14.408519 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:15:14.410481 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:14.414650 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:14.415973 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:14.417631 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:14.431895 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:14.432574 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:14.452984 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:15:14.525344 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 14:15:14.532407 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:14.533964 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:14.537403 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:14.543642 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:14.547176 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:14.547420 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:15:14.571295 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 14:15:14.575493 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:14.621311 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 14:15:14.631278 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:14.638310 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:14.643932 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:14.644784 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:14.647899 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:14.650103 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:15:14.652078 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 14:15:14.663729 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:14.711683 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:14.715445 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:14.723333 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:14.723913 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:15:14.729805 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 14:15:14.731564 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:14.743456 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:14.756170 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:14.794152 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:14.795853 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:14.799988 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:14.804255 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:14.813011 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:14.819406 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:14.838146 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:14.847273 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 14:15:14.888864 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:14.889663 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:14.898924 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:14.911015 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:14.911957 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:14.913763 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:14.919570 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:14.942085 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:15:14.954502 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:15:14.964207 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 14:15:15.001462 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:15.004168 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:15.010161 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:15.010823 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:15.023519 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:15.024859 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:15.045851 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:15.126770 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:15.126938 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:15.127713 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:15.129007 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:15.137465 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:15.148344 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:15.155994 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:15.219736 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 14:15:15.233716 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:15.236817 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:15.238682 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:15.242952 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:15.245235 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:15.251425 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:15.277487 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:15.330488 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:15.346593 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:15.349980 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:15.351382 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:15.352298 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:15.353282 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:15.353392 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:15.353767 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:15.388625 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 14:15:15.466982 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:15.469049 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:15.471221 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:15.471876 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:15.485745 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:15.493539 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:15.502240 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:15.574890 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:15:15.575225 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 14:15:15.577908 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:15.584877 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:15.591573 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:15.596290 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:15.622102 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:15.622716 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:15.623421 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:15.627605 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:15.632589 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:15.634662 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:15.640707 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:15.644482 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:15.669610 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:15.712730 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:15:15.735538 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:15.764925 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:15.768065 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:15.770380 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:15.772154 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:15.779995 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:15.791344 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:15.794482 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:15.819535 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:15:15.835388 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 14:15:15.835759 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:15.842159 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:15.842499 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:15.850945 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:15.851356 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:15.862188 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:15.904395 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240501 14:15:15.944742 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:15.945127 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:15.946811 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:15.955327 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:15.963857 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:15.964555 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:15.973122 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:16.128991 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:16.131119 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:16.131819 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:16.136598 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:16.137209 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:16.150447 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:16.161557 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:15:16.196218 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:16.248282 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:16.248340 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:16.250779 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:16.260165 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:16.264027 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:16.271420 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:16.297089 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:16.298105 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:16.307412 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:16.309639 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:16.312898 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:16.329795 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:16.333047 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:16.348284 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:15:16.364486 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:15:16.431082 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:16.436486 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:16.436554 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:16.440445 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:16.440924 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:16.441777 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:16.451463 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:16.544198 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:16.545843 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:16.546676 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:16.549178 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:16.550328 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:16.557346 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:16.567350 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:16.658740 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:16.661501 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:15:16.665633 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 14:15:16.666313 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:16.670020 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:16.672164 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:16.677778 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:16.681787 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:15:16.702929 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:15:16.728888 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:16.739543 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:16.739897 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:16.740566 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:16.742219 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:16.742367 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:16.744391 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:15:16.760535 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:16.830703 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:16.838614 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:16.840767 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:16.841094 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:16.842527 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:16.847214 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:16.861703 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:16.904698 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:16.905419 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:16.914986 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:16.916442 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:16.930476 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:16.930668 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:16.933112 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:15:17.002382 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:15:17.038960 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:15:17.058305 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:15:17.131239 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 14:15:17.153586 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:17.153925 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:17.156427 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:17.166141 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:15:17.174248 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 14:15:17.179118 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:17.183187 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:17.187579 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:17.258035 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:17.263084 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:17.270223 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:17.273571 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:17.277706 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:17.287238 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:17.305567 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:15:17.315260 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 14:15:17.324100 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:17.329493 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:17.333015 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:17.340157 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:17.340190 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:17.346256 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:17.360378 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:15:17.364001 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:17.435458 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:17.438932 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:17.443352 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:17.445159 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:17.445704 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:17.447759 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:17.457813 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:15:17.493249 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:15:17.511437 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 14:15:17.542567 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:17.542980 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:17.548708 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:17.553875 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:17.554095 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:17.571645 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:17.573040 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:17.652390 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:17.655015 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:17.658733 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:17.663533 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:17.673278 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:17.674885 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:17.682669 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:17.694191 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 14:15:17.703461 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:17.712083 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:17.713057 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:17.717099 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:17.727210 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:17.736708 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:17.745405 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:17.838294 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:17.840052 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:17.844305 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:17.852365 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:17.863848 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:17.869143 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:17.875319 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:17.878544 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:17.921922 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:17.926611 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:17.928519 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:17.929358 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:17.935191 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:17.943192 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:17.951347 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:18.094822 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 14:15:18.097596 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:15:18.124032 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 14:15:18.175195 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:18.175634 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:18.178496 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:18.178980 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:18.183840 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:18.185263 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:18.194011 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:18.297814 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 14:15:18.299345 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:18.307114 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:18.310374 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:18.311030 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:18.320197 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:18.327759 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:18.332299 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:18.333930 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:18.339211 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:18.340595 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:18.342535 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:18.359786 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:18.369107 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:18.380648 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:15:18.401152 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:18.473523 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:18.473846 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:18.479261 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:18.480907 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:18.485419 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:18.493458 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:18.511801 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:15:18.513334 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 14:15:18.584733 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:18.585155 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:18.587013 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:18.592152 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:18.593602 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:18.607674 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:18.627758 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:18.710765 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:18.715021 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:18.722105 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:18.726147 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:18.732421 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:18.732511 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:18.751811 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:18.752121 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:18.769333 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:18.773006 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:18.781229 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:18.787426 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:18.800516 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:18.802193 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:18.901526 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:18.908252 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:18.908679 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:18.913566 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:18.923178 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:18.925151 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:18.926615 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:18.934015 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:18.939200 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:15:18.943512 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:18.951300 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:18.951817 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:18.966764 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:18.985324 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:18.991716 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:19.225761 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 14:15:19.264021 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:19.267992 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:19.275379 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:19.276296 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:19.276917 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:19.286592 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:19.297149 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:15:19.304764 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 14:15:19.336340 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:19.337211 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:19.340247 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:19.348724 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:19.352638 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 14:15:19.353274 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:19.353435 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:19.376850 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:15:19.409736 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:15:19.430929 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:19.436203 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:19.438354 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:15:19.439217 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:19.439579 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:19.440974 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:15:19.447342 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 14:15:19.457298 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:19.458012 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 14:15:19.462572 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:19.528854 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:19.530495 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:19.538453 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:19.541504 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:19.550060 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:19.553108 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:19.556942 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:15:19.590785 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 14:15:19.649212 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:19.649375 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:19.650983 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:19.651168 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:19.652966 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:19.663540 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:19.680159 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:19.712994 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 14:15:19.777329 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:19.779016 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:19.786387 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:19.803272 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:19.803658 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:19.803803 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:19.804183 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 14:15:19.804518 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:19.852536 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:19.854285 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:19.855064 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:19.857733 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:19.858763 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:19.861981 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:19.884112 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:19.935297 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:15:19.955597 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:19.973042 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:19.973778 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:19.974169 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:15:19.975739 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 14:15:19.976645 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:19.978336 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:19.992843 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:20.001471 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:20.019884 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:20.023173 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:20.024978 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:20.026158 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:20.028308 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:20.032017 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:20.043777 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:15:20.297504 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:15:20.325922 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 14:15:20.354269 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:20.358003 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:20.359089 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:20.375138 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:20.378712 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:20.382081 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:20.388227 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:20.399798 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:20.400311 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:20.404892 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:20.404927 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 14:15:20.414362 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:20.416776 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:20.418459 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:20.444574 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:20.531831 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 14:15:20.536316 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:20.537067 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:20.539420 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:20.541646 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:20.548403 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:20.549948 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:20.560024 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:20.562968 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:20.563546 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:20.595749 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:20.602239 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:20.605284 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:20.606292 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:20.607013 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:20.620203 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:20.695256 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:20.698649 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:20.703747 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:20.713080 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:20.721081 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:20.721504 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:20.722872 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:20.814072 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:20.816596 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:20.821507 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:20.822137 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:20.828564 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:20.835429 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:20.844416 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:15:20.934193 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240501 14:15:20.941283 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:20.943303 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:20.950949 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:20.951920 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:20.960021 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:20.968603 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:20.971006 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:20.971621 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:20.976130 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:20.977528 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:20.986402 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:20.992411 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:20.993600 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:20.996513 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:21.033118 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:15:21.037917 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:21.102021 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:21.102298 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:21.113641 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:21.117967 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:21.119343 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:21.119411 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:21.122897 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:21.311494 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:21.312079 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:21.320979 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:21.327822 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:21.328713 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:21.331169 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:21.346179 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:21.376184 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:15:21.430446 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:21.438175 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:21.444949 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:21.451491 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:21.452795 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:21.456633 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:21.457444 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:21.492817 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:21.500926 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:21.501991 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:21.503945 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:21.514174 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:21.516993 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:21.528995 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:15:21.598209 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:21.605654 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:21.612870 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:21.614914 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:21.616981 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:21.627679 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:21.630636 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:21.655025 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:21.656105 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:21.663520 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:21.670884 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:21.674368 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:21.674609 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:21.679791 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:21.694988 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:21.717473 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 14:15:21.787608 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:21.789423 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:21.791523 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:21.791687 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:21.800457 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:21.802424 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:21.804955 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:21.861217 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 14:15:21.901554 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:21.906599 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:21.909593 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:15:21.919796 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 14:15:21.920290 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:21.920548 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:21.941426 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:21.945555 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:21.947441 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:15:22.012212 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:22.014575 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:22.018782 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:22.021677 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:22.022218 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:22.022723 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:22.029652 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:22.046783 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:22.053205 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:15:22.116406 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 14:15:22.127323 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:22.131301 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:22.131764 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:22.141230 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:22.141429 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:22.150974 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:22.152896 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:22.216931 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 14:15:22.229420 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:22.235219 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:22.238420 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:22.243942 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:22.247068 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:22.252614 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:22.253053 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:22.342909 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:22.343622 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:22.349977 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:22.351156 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:22.357920 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:22.360430 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:22.365057 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:15:22.380236 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:15:22.421139 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 14:15:22.453330 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:22.460793 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:22.462733 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:22.471829 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:22.479795 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:15:22.491323 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:22.491508 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:22.492511 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:22.500556 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:22.500756 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:22.504714 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:22.506381 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:22.532325 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:22.541884 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:22.542304 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:15:22.544211 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 14:15:22.644766 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:22.647378 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:22.655092 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:22.664902 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:22.666241 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:22.666476 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:22.668483 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:22.668816 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:22.669052 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:22.680974 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:22.688648 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:22.701923 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:22.713258 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:22.723721 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:15:22.752694 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 14:15:22.792382 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:22.793164 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:22.794600 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:22.799870 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:22.807971 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:22.818128 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:22.828917 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:22.934821 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:15:22.956511 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:15:22.957531 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:15:22.992029 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:23.016033 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:23.016696 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:23.020779 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:23.021195 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:23.035727 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:23.038483 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:23.045408 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:23.147540 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:23.147881 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:23.152334 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:23.159628 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:23.167199 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:23.176470 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:23.177994 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:23.179873 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:23.182889 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:23.189572 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:23.197294 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:23.198946 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:23.199534 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:23.200711 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:15:23.273259 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 14:15:23.312232 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:23.313256 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:23.317466 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:23.320783 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:23.322579 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:23.328835 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:23.347389 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:15:23.378945 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 14:15:23.428740 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:23.429832 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:23.437741 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:23.441690 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:23.447813 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:23.456176 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:23.470183 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:15:23.497277 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:23.548686 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:23.558363 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:23.568181 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:23.570755 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:23.571106 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:23.571820 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:23.593204 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:23.603497 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:23.613006 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:23.621088 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:23.623857 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:23.624303 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:23.641311 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:23.648999 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:23.741624 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:23.742234 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:23.754138 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:23.758414 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:23.758985 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:23.765343 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:23.771514 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:23.835636 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:23.835860 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:23.840579 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:23.841267 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:23.841346 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:23.853489 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:23.863090 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:23.921494 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:24.081454 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:24.082942 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:24.089197 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:24.090878 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:24.093868 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:24.100570 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:24.109301 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 14:15:24.115232 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:24.212152 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:24.214803 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:24.214825 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:24.215791 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:24.216482 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:15:24.223837 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 14:15:24.237287 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:24.241454 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:24.241930 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:24.245571 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:24.254189 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:24.257591 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:24.257865 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:24.257980 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:24.260368 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:24.305330 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:15:24.324527 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:15:24.325240 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:24.379948 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:24.380245 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:24.381649 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:24.382074 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:24.395725 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:24.402710 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:24.403864 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:15:24.459264 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:15:24.490656 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:24.490928 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:24.494611 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:24.496923 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:24.503213 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:24.504377 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:24.512471 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:15:24.538360 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:15:24.543089 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 14:15:24.602663 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:24.609182 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:24.613235 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:24.614650 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:24.620061 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:24.626490 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:24.627126 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:15:24.644444 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 14:15:24.680161 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:24.684218 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:24.685748 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:24.688269 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:24.701045 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:24.701921 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:24.702145 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:24.765725 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:24.770815 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:24.775194 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:24.775442 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:24.786660 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:24.808373 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:24.808491 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:24.864219 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:24.865095 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:24.866389 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:24.867256 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:24.867251 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:24.867972 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:24.868499 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:24.893221 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:15:24.964531 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:15:25.000609 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:15:25.070461 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 14:15:25.098169 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:25.099011 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:25.101150 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:25.108482 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:25.109951 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:25.113711 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:25.128134 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:25.163648 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 14:15:25.217883 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:25.221166 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:25.221421 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:25.233574 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:25.234644 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:25.235292 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:25.236042 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:25.244057 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:25.250717 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:25.253360 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:25.264581 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:25.277112 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:25.287493 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:25.304847 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:25.384649 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:25.386225 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:25.391887 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:25.397027 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:25.401157 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:25.408073 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:25.428153 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:25.454243 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:15:25.461199 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:15:25.468626 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 14:15:25.507872 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:15:25.508028 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 14:15:25.508715 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:25.515983 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:25.519464 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:25.520977 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:25.525755 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:25.528420 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:25.632925 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:25.641429 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:25.642261 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:25.652928 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:25.653663 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:25.657485 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:25.664005 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:25.668004 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:25.682077 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:25.690935 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:25.694382 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:25.695673 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:25.711508 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:25.713647 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:25.834520 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:25.837049 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:25.844862 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:25.847967 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:25.849721 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:25.851847 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:25.856737 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:25.858144 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:25.864207 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:25.872395 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:25.879200 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:25.892884 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:25.908331 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:25.912402 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:25.937757 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 14:15:25.947302 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:15:26.009902 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 14:15:26.220827 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:26.221326 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:26.226938 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:26.227871 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:26.233683 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:26.254590 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:26.266929 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:26.271027 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:26.272825 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:26.280335 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:26.281311 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:26.282984 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:26.285841 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:26.307366 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:15:26.375698 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 14:15:26.407754 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:26.409449 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:26.418524 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:26.424952 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:15:26.430437 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 14:15:26.434621 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:26.442582 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:26.449913 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:26.464058 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:26.490805 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:26.492149 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:26.502424 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:26.503165 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:26.514437 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:26.529695 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:15:26.530524 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:26.606859 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:26.607501 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:26.609719 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:26.617192 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:26.617619 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:26.623904 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:26.624421 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:26.689977 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:15:26.701861 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:15:26.729048 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:26.731213 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:26.731345 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:26.733861 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:26.736774 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:26.754205 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:26.756389 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:26.814692 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:26.817727 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:26.818778 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:26.822259 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:26.834815 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:26.844310 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:26.859323 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:26.907579 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:26.909921 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:26.916800 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:26.926584 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:26.930171 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:26.931407 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:26.938303 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:26.986377 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:26.990278 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:15:26.992055 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:26.993052 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:26.994350 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:27.001916 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:27.005075 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:27.018631 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:27.057709 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:15:27.111517 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 14:15:27.243710 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:27.245083 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:27.247535 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:27.247541 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:27.252146 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:27.256132 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:27.260234 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:27.357256 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:27.359429 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:15:27.362287 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 14:15:27.367923 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:27.371220 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:27.379377 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:27.387725 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 14:15:27.388725 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:27.398900 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:15:27.419349 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:27.427488 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:27.431413 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:27.442679 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:27.445555 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 14:15:27.456611 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:27.459398 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:27.467872 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:27.470360 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:15:27.524821 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:15:27.538143 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:15:27.539467 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 14:15:27.544749 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:27.551113 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:27.555538 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:27.567752 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:27.569111 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:27.571439 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:27.615232 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:27.621546 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:27.626353 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:27.636085 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:27.644500 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:27.650167 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:27.652211 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:27.655148 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 14:15:27.747253 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:27.748651 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:27.750838 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:27.754246 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:27.761519 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:27.766702 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:27.767426 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:15:27.846031 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 14:15:27.868232 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:27.873546 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:27.874701 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:27.884730 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:27.888235 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:27.892362 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:27.910465 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:27.912448 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:27.921815 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:27.931389 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:27.931435 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:27.933462 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:27.941823 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:27.943878 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:27.945777 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:15:27.982607 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 14:15:28.049863 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:28.053205 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:28.054783 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:28.058620 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:28.071110 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:28.071820 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:28.081180 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:15:28.096356 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 14:15:28.254667 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:28.254953 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:28.255645 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:28.263597 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:28.264891 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:28.268538 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:28.297118 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:28.370501 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:28.370976 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:28.379233 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:28.380024 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:28.387161 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:15:28.409130 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 14:15:28.411890 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:28.419791 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:28.431269 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:28.433858 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:28.435293 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:28.453039 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:15:28.453965 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 14:15:28.464417 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:28.475749 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:28.478303 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:28.484216 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:28.536785 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:28.538991 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:28.544991 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:28.555189 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:28.558646 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:28.567783 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:28.572732 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:28.609117 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:28.612701 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:28.614857 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:28.626528 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:28.627454 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:28.642546 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:28.643909 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:28.723464 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:28.728840 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:28.731016 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:28.732719 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:28.733060 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:28.739192 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:28.753135 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:28.833463 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:28.837210 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:28.841187 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:28.849882 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:28.851528 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:28.859033 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:15:28.862227 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 14:15:28.863391 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:28.876745 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:28.918097 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:28.926540 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:28.927933 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:28.928350 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:28.929944 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:28.935807 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:28.949738 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:29.030484 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:29.030822 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:29.035568 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:29.036850 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:29.040953 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:29.053880 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:29.055158 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:15:29.075562 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 14:15:29.167372 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:29.168963 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:29.175200 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:29.176357 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:29.180645 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 14:15:29.183136 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:29.188098 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:29.194268 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:29.288444 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:29.290431 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:29.295244 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:29.303593 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:29.304600 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:29.311905 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:29.316082 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:15:29.351655 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 14:15:29.380767 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:29.383627 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:29.387511 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:29.387593 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:29.402007 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:29.419512 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:29.425772 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:15:29.454689 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:15:29.462093 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 14:15:29.462889 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:29.469216 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:29.469249 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:29.482518 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:29.488525 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:29.497692 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:29.501097 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:29.503154 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:15:29.535915 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 14:15:29.542270 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:29.549036 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:29.550266 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:29.550276 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:29.552577 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:29.560717 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:29.564816 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:29.647608 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:29.653268 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:29.653739 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:29.656360 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:29.661232 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:29.666039 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:29.672152 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:15:29.745086 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 14:15:29.755954 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:29.757051 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:29.759106 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:29.763345 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:29.763805 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:29.766871 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:29.767186 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:29.866531 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:29.867523 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:29.870726 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:29.871500 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:29.875090 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:29.881567 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:29.889276 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:29.899832 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 14:15:29.966666 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:29.968735 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:29.977874 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:15:29.979820 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:29.982271 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:29.986120 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:29.986371 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:29.989116 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:15:30.031945 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:15:30.076148 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 14:15:30.082278 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:30.092523 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:30.102769 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:30.104440 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:30.112489 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:30.112728 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:30.112743 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:15:30.180794 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:15:30.196806 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 14:15:30.227864 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:30.228938 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:30.229624 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:30.235198 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:30.243163 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:30.245380 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:30.250841 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:30.341033 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:30.346298 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:30.352391 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:30.353276 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:30.353607 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:30.375227 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:30.378090 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:30.382246 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:30.404773 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:30.409498 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:30.409512 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:30.411152 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:30.422051 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:30.426182 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:30.441515 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:30.478725 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 14:15:30.517717 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:30.518568 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:30.520057 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:30.522557 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:30.529268 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:30.532030 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:30.532934 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:30.604938 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:30.607957 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:30.609839 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:30.611008 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:30.624212 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:30.627290 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 14:15:30.628079 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:30.628649 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:30.710608 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:30.714601 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:30.716445 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:30.726770 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:30.736912 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:30.741508 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:30.741941 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:30.826815 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:30.831892 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:30.834442 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:30.840404 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:30.843472 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:30.848994 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:30.852152 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:30.936651 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:30.938987 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:30.941246 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:15:30.941437 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:15:30.945577 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240501 14:15:30.948120 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:30.956635 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:30.957522 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:30.960860 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:15:30.983678 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:31.040709 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:31.041252 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:31.042678 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:31.046300 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:31.066366 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:31.067595 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:31.075475 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:31.156939 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:31.156929 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:31.157917 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:31.166800 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:31.178809 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:31.178890 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:31.181340 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:31.289677 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:31.290668 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:31.294044 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:31.294414 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:31.303934 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:31.307262 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:31.307710 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:31.314088 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:31.314368 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:31.319634 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:31.324119 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:31.340211 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:31.343758 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:31.347797 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 14:15:31.359411 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:31.450785 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:15:31.453622 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 14:15:31.455096 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:31.464167 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:31.465966 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:31.476562 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:31.479591 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:31.483656 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:31.483963 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:31.484011 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:31.484027 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:31.491217 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:31.514186 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:31.525036 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:15:31.537984 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:31.540796 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:31.613006 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:31.619025 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:31.626747 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:31.627983 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:31.633105 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:31.634452 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:31.635349 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:31.718400 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 14:15:31.803789 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:31.807241 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:31.814522 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:31.817171 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:31.818799 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 14:15:31.826648 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:31.828583 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:31.829058 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:31.925665 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:31.926342 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:31.933326 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:31.937587 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:31.941706 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:31.943392 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:31.958738 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:31.999742 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:32.005206 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:32.008031 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:32.009624 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:32.009685 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:32.011018 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:15:32.026907 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:15:32.033615 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:15:32.118903 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:32.119318 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:32.127233 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:32.128226 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:32.131110 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:32.134039 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:32.135164 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:32.136487 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:32.225564 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 14:15:32.241415 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:32.241688 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:32.241869 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:32.242733 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:32.252051 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:32.268991 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:32.275514 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:32.317309 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:32.326804 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:32.328497 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:32.332495 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:32.339293 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:32.341850 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:32.342283 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:15:32.402316 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 14:15:32.427192 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:32.438056 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:32.440491 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:32.440550 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:32.441962 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:32.444944 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:32.449245 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:15:32.494868 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 14:15:32.506492 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:32.510560 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:32.514245 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:32.521749 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:32.523124 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:32.529757 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:32.538328 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:32.595229 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:32.600574 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:32.602617 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:32.605259 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:32.615165 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:15:32.624313 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:32.628058 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:15:32.630220 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 14:15:32.637938 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:32.639847 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:15:32.749373 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:15:32.781351 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:15:32.831449 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:32.834208 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:32.835384 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:32.836006 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:32.836218 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:32.838647 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:32.841792 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:15:32.856611 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 14:15:32.953706 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:32.958737 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:32.960697 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:32.964221 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:32.968520 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:32.977294 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:32.978212 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:32.980023 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:32.984800 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:32.985692 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:33.000416 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:15:33.015645 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:15:33.016819 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:33.017714 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:33.018702 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:33.040141 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:33.120726 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:33.123271 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:33.125548 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:33.125838 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:33.145946 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:33.146682 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:33.148432 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:33.234460 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:33.236688 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:33.241501 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:33.250694 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:33.252862 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:33.253592 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:33.254004 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:15:33.338083 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 14:15:33.355820 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:33.359345 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:33.361683 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:33.365958 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:33.376196 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:33.376657 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:33.388986 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:33.411988 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:33.412581 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:33.416105 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:33.422276 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:33.423729 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:33.427799 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:33.439949 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:33.525853 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 14:15:33.550897 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:33.550908 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:33.562422 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:33.563258 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:33.565599 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:33.572012 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:33.579252 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:33.582836 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:33.583895 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:33.590468 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:33.605677 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:33.612972 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:33.616359 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:33.629474 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:33.649427 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:15:33.654383 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 14:15:33.942998 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:15:33.944629 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 14:15:33.945045 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:33.945077 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:33.948572 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:33.951380 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:33.960471 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:33.967451 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:33.990438 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:33.996409 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:33.998643 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:34.010082 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:34.011812 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:34.013605 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 14:15:34.013826 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:34.020450 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:15:34.039798 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 14:15:34.105476 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:34.110122 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:34.116856 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:34.117611 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:34.122262 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:34.130478 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:34.134322 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:34.151710 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 14:15:34.171497 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:34.171922 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:34.175205 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:34.180883 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:34.189494 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:34.193486 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:34.204077 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:34.232362 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:34.289891 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:34.293480 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:34.293915 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:34.294131 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:34.294528 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:34.319229 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:34.320365 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:34.404160 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:34.406144 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:34.414749 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:34.417018 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:34.421097 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:34.425884 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:34.468616 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:34.481703 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:34.486924 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:34.487879 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:34.499161 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:34.500386 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:34.501363 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:34.513819 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:34.580624 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:34.584944 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:34.588326 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:34.591347 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:34.596716 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:34.603206 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:34.603710 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:34.659790 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:34.662119 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:34.666085 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 14:15:34.668248 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:34.668574 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:34.669951 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:34.677846 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:34.691536 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:15:34.697641 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:15:34.720008 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 14:15:34.766757 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:15:34.767985 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:15:34.775791 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:15:34.778299 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 14:15:34.910012 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:34.911679 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:34.920430 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:34.926359 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:34.926574 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:34.936797 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:34.945969 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:35.022706 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:35.024716 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:35.028563 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:35.029325 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:35.043556 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:35.045437 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:35.059402 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:15:35.075865 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 14:15:35.111124 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:35.111852 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:35.114109 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:35.124739 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:35.126755 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:35.127384 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:35.139039 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:15:35.152807 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 14:15:35.204494 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:35.204737 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:15:35.206599 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:35.208441 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:35.216550 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:35.228541 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:35.235726 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:15:35.240350 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 14:15:35.243310 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:35.291640 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:35.294577 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:35.297744 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:35.302179 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:35.314472 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:35.314527 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:35.332968 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:35.411087 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:35.415161 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:35.416733 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:35.427281 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:35.428829 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:35.429081 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:35.435478 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:35.548768 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:35.550555 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:35.553050 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:35.557965 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:35.558665 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:35.567267 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:35.571245 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:35.572443 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:35.576476 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:35.579222 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:35.587201 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:35.592584 19486 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:35.592631 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:35.608023 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:35.620795 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:15:35.646924 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 14:15:35.707698 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:35.712414 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:35.713984 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:35.716198 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:35.734508 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:35.752324 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:35.754698 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:35.890637 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 14:15:35.927242 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:35.927467 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:35.935292 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:35.935757 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:35.943617 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:35.944291 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:35.956192 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:15:35.986948 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 72 similar messages]
I20240501 14:15:36.059453 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:36.064051 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:36.070899 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:36.073266 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:15:36.074541 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 14:15:36.075551 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:36.080897 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:36.083799 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:36.083802 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:36.084300 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:36.093737 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:36.103178 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:36.105193 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:36.115187 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:36.117743 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:15:36.148487 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:36.230216 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:36.239603 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:36.241340 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:36.242280 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:36.245254 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:36.248251 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:36.265587 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:15:36.267753 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 14:15:36.268087 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:36.272755 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:36.288918 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:36.298266 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:36.305132 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:36.307325 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:36.308043 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:36.398550 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:36.403931 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:36.404515 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:36.411149 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:36.415508 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:36.420300 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:36.424822 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:36.432127 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:15:36.509147 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 14:15:36.513971 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:36.514420 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:36.518587 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:36.526028 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:36.529044 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:36.533519 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:36.547850 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:15:36.569514 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:15:36.582731 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:36.619339 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:36.621011 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:36.622416 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:36.626554 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:36.631850 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:36.633953 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:36.647630 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:36.731124 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:36.732067 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:36.732971 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:36.733044 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:36.735602 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:36.741400 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:36.763142 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:36.836021 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:36.839020 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:36.845098 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:36.850659 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:36.855105 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:36.856626 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:36.869117 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:36.958578 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:36.959136 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:36.963093 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:36.964797 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:36.965605 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:36.967485 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:36.975945 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:37.079921 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:37.079964 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:37.081467 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:37.099825 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:37.101619 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:37.102653 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:37.102932 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:37.111196 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:37.112380 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:15:37.119454 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 14:15:37.124794 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:37.135540 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:37.146067 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:37.153383 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 14:15:37.155709 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:37.155964 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:37.186515 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:15:37.200680 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:37.263473 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:37.267437 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:37.282173 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:37.289894 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:37.291093 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:37.295403 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:37.307199 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:37.311493 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:37.318559 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:37.323371 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:37.329295 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:37.334235 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:37.339640 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:37.347788 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:37.434623 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:37.434823 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:37.435693 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:37.437218 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:37.441342 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:37.448446 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:37.467638 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:15:37.508911 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:15:37.509658 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:15:37.511224 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 14:15:37.646528 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:37.647023 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:37.648543 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:37.649382 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:37.655274 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:37.659646 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:37.663252 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:15:37.694761 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:15:37.734862 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:15:37.758675 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 14:15:37.776039 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:37.784035 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:37.785406 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:37.793720 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:37.798403 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:37.801594 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:37.802845 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:37.808954 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:37.820044 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:37.830345 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:37.848114 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:15:37.850488 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:15:37.866180 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:37.872614 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:37.873036 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:37.962980 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:37.965489 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:37.967525 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:37.969885 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:37.972769 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:37.985352 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:37.988512 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:38.080217 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:38.080641 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:38.088696 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:38.088977 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:38.089691 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:38.108079 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:38.117664 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:38.135335 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 14:15:38.189122 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:38.190048 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:38.191017 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:38.201365 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:38.204294 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:38.232710 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:38.236986 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:38.258771 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:38.259191 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:38.260109 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:38.265034 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:38.273200 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:38.288445 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:38.293582 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:38.295166 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:15:38.302870 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:15:38.311965 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 14:15:38.379981 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:38.380774 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:38.382401 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:38.385906 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:38.388623 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:38.396066 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:38.405321 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:38.454803 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:38.458544 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:38.461359 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:38.462844 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:38.465081 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:38.465276 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:38.489578 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:15:38.597352 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 14:15:38.723352 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:38.723361 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:38.736351 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:38.737823 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:38.742240 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:38.742883 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:38.759277 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:15:38.762384 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:38.849336 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:38.851469 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:38.857354 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:38.860827 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:38.866989 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:38.886372 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:38.888747 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:38.889043 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:38.892622 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:38.901417 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:38.903930 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:38.923192 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:38.924480 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 14:15:38.933367 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:38.941423 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:15:39.005666 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 14:15:39.028234 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:39.029759 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:39.029912 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:39.031361 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:39.033959 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:39.035451 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:39.051179 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:39.067569 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 14:15:39.139982 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:39.140406 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:39.142741 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:39.145507 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:39.146023 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:39.161780 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:39.162050 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:39.174310 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:39.254623 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:39.259148 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:39.263185 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:39.264988 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:39.275789 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:39.284287 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:39.300163 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:39.326800 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:39.334483 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:39.337198 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:39.339402 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:39.345196 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:39.365957 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:39.366227 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:39.393986 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 14:15:39.455224 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:39.460376 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:15:39.463131 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 14:15:39.464480 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:39.468673 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:39.471199 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:39.487495 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:39.489878 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:39.491369 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:15:39.497459 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 14:15:39.499903 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:39.517817 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:39.519219 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:39.519507 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:39.519979 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:39.526103 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:39.767118 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:15:39.767750 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:39.801846 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:39.805795 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:39.806624 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:39.807086 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:39.824106 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:39.830770 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:39.845188 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:39.861117 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:15:39.865991 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:15:39.871971 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 14:15:39.880445 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:39.881471 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:39.885358 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:39.886129 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:39.889570 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:39.891271 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:39.934501 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:39.961268 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 14:15:39.971994 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:39.974149 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:39.977298 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:39.986078 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:39.988442 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:39.991318 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:39.991627 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:40.064970 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:40.065541 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:40.072082 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:40.073558 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:40.075625 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:40.093215 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:40.096458 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:40.173599 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:15:40.176983 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240501 14:15:40.177815 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:40.178345 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:40.179390 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:40.186935 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:40.190474 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:40.201876 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:40.304587 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:40.308130 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:40.315428 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:40.317111 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:40.323402 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:40.324620 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:40.329320 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:40.336175 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:40.345317 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:40.348981 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:40.355487 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:15:40.372016 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:40.376119 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:40.382954 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:40.384696 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:15:40.448573 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:15:40.493165 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 14:15:40.513535 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:40.513541 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:40.530622 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:40.533562 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:40.536370 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:40.543202 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:40.545585 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:40.550877 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:40.563498 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:40.565611 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:40.571277 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:40.580610 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:40.584779 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:40.590936 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:15:40.756187 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 14:15:40.891613 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:40.896543 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:40.901059 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:40.906005 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:15:40.912061 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 14:15:40.916033 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:40.916797 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:40.917477 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:40.945214 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:40.946095 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:40.951505 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:40.955802 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:40.961122 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:40.961809 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:40.974134 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:15:40.990374 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:15:41.011988 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240501 14:15:41.085098 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:41.086333 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:41.090425 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:41.092926 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:41.093415 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:41.102113 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:41.109001 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:41.110414 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:41.111310 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:41.113377 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:41.121469 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:41.131531 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:41.150123 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:41.161664 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:41.242172 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:41.245889 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:41.247963 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:41.255421 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:41.267997 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:41.282068 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:41.288564 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:41.367404 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:41.368432 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:41.369374 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:41.376814 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:41.379201 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:41.385444 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:41.386241 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:41.403765 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:15:41.462193 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:15:41.487242 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:41.498142 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:41.500806 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:41.502105 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:41.506412 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:41.513736 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:41.517467 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:41.523329 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:41.525141 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:41.527465 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:41.529105 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:41.538935 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:41.566468 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:41.567416 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:41.571661 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:15:41.579113 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 14:15:41.653858 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:41.656612 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:41.657460 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:41.661087 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:41.665674 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:41.665779 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:41.668560 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:15:41.817911 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:15:41.823716 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:15:41.826920 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 14:15:41.858065 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:41.859939 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:41.860807 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:41.861709 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:41.862484 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:41.884697 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:41.887310 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:41.962855 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:15:41.966603 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:41.966671 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:41.979497 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:41.983736 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:41.985987 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:42.000752 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:42.018823 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:42.022825 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:42.023983 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:42.026847 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:42.038050 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:42.053910 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:42.056974 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:42.061374 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:42.151218 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:42.152431 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:42.159185 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:42.162379 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:42.166096 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:42.171870 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:42.172179 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:15:42.193188 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 14:15:42.209266 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:42.209525 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:42.218566 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:42.221813 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:42.222200 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:42.223737 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:42.224403 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:42.326139 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:42.328747 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:42.337205 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:42.338903 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:42.340559 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:42.347172 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:42.351233 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:15:42.403924 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 14:15:42.438182 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:42.438274 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:42.439258 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:42.443225 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:42.448668 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:42.461678 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:42.470054 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:42.498394 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:15:42.514971 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:42.523759 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:42.527426 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:42.531556 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:42.532544 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:42.533250 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:42.541620 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:42.543951 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:15:42.605276 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 14:15:42.624866 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:42.626857 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:42.629727 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:42.633030 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:42.636305 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:42.638479 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:42.657740 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:42.753440 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:42.757790 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:42.761983 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:42.762471 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:42.762686 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:42.763307 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:42.766821 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:42.869599 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:42.871711 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:42.876211 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:42.883309 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:42.883860 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:15:42.884384 21333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:15:42.887238 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:42.900475 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:42.998646 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:43.001441 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:43.003774 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:43.004115 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:43.010742 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:43.016839 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:43.018594 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:15:43.019857 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 14:15:43.026407 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:43.028192 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 14:15:43.050618 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:43.060959 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:43.062932 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:43.071278 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:43.073500 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:43.079574 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:15:43.097409 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:15:43.137856 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:43.153663 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:43.154403 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:43.160794 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:43.162708 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:43.169242 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:43.169497 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:43.181560 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:43.201371 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:43.202453 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:43.205678 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:43.211532 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:43.211699 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:43.224475 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:43.229456 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:43.284147 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:15:43.284694 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 14:15:43.329061 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:43.330055 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:43.333369 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:43.335937 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:43.344851 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:43.346406 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:43.369225 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:43.529834 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:43.531818 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:43.535210 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:15:43.536525 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:43.537874 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:43.538358 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:43.543213 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:43.565661 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:43.651551 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:43.657635 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:43.666174 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:43.671947 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:43.673022 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:43.673579 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:43.688303 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:43.695719 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:43.703967 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:43.709904 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:43.716990 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:43.731194 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:43.732957 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:43.739498 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:43.840493 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:43.844026 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:43.849743 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:43.851217 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:43.851754 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:43.854099 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:43.868064 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:15:43.890209 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 14:15:43.891085 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 14:15:43.958719 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:43.963215 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:43.966954 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:43.967161 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:43.970917 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:43.975172 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:43.975646 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:15:43.999141 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:44.041025 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:44.047165 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:44.049125 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:44.052311 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:44.053907 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:15:44.060647 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 14:15:44.061205 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:44.078783 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:44.093268 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 14:15:44.124526 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:44.133263 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:44.135615 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:44.140898 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:44.142534 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:44.143659 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:44.166641 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:15:44.180688 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 14:15:44.208734 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:44.211791 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:44.215078 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:44.217094 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:44.222309 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:44.245013 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:44.249778 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:44.297664 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:44.298777 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:44.304728 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:44.307018 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:44.309697 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:44.313670 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:44.332162 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:15:44.430358 19572 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:15:44.460857 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:15:44.532603 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 14:15:44.547667 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:44.551227 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:44.551031 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:44.556354 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:44.556874 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:44.559463 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:44.574086 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:15:44.600436 19572 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 14:15:44.658226 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:44.661213 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:44.671203 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:44.681185 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:44.681600 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:44.686096 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:44.688798 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:44.702574 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:44.717454 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:44.717566 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:44.717831 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:44.728413 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:44.732388 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:44.739507 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:44.796681 19310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:15:44.797421 19310 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 14:15:44.830264 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:44.831172 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:44.836687 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:44.840031 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:44.850587 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:44.852893 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:44.855527 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:44.923004 19572 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:15:44.930987 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:44.943672 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:44.944413 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:44.944865 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:44.948318 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:44.951519 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:44.952414 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:44.971333 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:45.062681 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:45.063604 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:45.071280 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:45.073930 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:45.087504 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:45.100236 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:45.108160 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:45.115886 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:45.123782 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:45.127633 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:45.127710 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:45.136394 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:45.151387 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:45.154384 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:45.165467 21333 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 14:15:45.235366 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:45.238083 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:45.239701 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:45.255777 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:45.266440 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:45.267413 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:45.271064 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 14:15:45.287983 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:15:45.291304 19441 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 14:15:45.297761 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:45.301648 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:45.316633 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:45.317205 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:45.318434 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:45.323568 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:45.333524 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:45.409552 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:15:45.541348 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 14:15:45.596551 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:45.603222 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:45.608397 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:45.610666 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:45.610937 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:45.611685 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:45.627763 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:45.679129 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:45.679379 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:45.685254 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:45.685642 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:45.686216 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:45.706566 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:45.726408 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:45.768095 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:45.771790 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:45.773383 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:45.775790 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:45.775971 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:45.806855 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:45.809330 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:45.845773 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 14:15:45.869367 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:45.870283 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:45.872483 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:45.873777 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:45.880870 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:45.884987 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:45.895087 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:45.927165 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 14:15:45.976488 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:45.977993 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:15:45.979907 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:45.983794 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:45.988337 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:45.988430 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:46.004040 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:46.011246 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:46.101872 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:15:46.110355 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 67 similar messages]
I20240501 14:15:46.111129 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:46.112211 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:46.114499 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:46.129033 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:46.129297 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:46.134613 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:46.150075 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:46.152055 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:46.157194 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:46.161103 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:46.175892 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:46.183512 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:46.204603 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:46.253566 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:46.253834 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:46.254540 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:46.260083 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:46.260157 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:46.276845 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:46.277709 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:46.333518 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:46.334972 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:46.343034 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:46.345207 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:46.348610 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:46.351233 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:46.359566 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:46.360285 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 14:15:46.543336 19310 consensus_peers.cc:462] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:15:46.558099 19572 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 14:15:46.577181 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:46.580497 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:46.580890 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:46.581591 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:46.584450 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:46.591266 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:46.613219 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:46.669631 19572 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 14:15:46.689008 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:46.692555 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:15:46.693758 19572 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 14:15:46.694959 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:46.699342 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:46.699560 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:46.715747 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:46.726392 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:46.749572 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:46.759202 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:46.760423 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:46.761044 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:46.772933 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:46.774168 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:46.778425 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:46.850325 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:46.857311 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:46.857396 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:46.857800 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:46.858027 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:46.860152 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:46.867574 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:46.937461 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:46.937578 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:46.940659 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:46.943804 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:46.944577 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:46.948184 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:46.965277 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:46.983628 19572 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240501 14:15:47.040620 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:47.042500 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:47.044638 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:47.044683 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:47.060755 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:47.061471 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:47.067618 11111 raft_consensus.cc:998] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d: Attempting to remove follower e6246b183293412494919f1e7efc9448 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e6246b183293412494919f1e7efc9448 for more than 300 seconds (300.093s)
I20240501 14:15:47.067965 11111 consensus_queue.cc:237] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:15:47.069208 21137 raft_consensus.cc:1261] T f69020bb6c244848a6b0a50fd2b4f3eb P 5bf81f3d986e428190d3f7727f36d96c [term 2 FOLLOWER]: Refusing update from remote peer 43ce4a4c954f4a5f80954b38933c189d: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 14:15:47.070351 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:47.070724 11111 consensus_queue.cc:1035] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:15:47.077368 6863 raft_consensus.cc:998] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc: Attempting to remove follower e6246b183293412494919f1e7efc9448 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e6246b183293412494919f1e7efc9448 for more than 300 seconds (300.421s)
I20240501 14:15:47.077641 6863 consensus_queue.cc:237] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:15:47.078190 11111 raft_consensus.cc:2932] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:15:47.079053 21522 raft_consensus.cc:1261] T 410a7dd0dd734684a82875912966657d P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:15:47.080356 21137 raft_consensus.cc:2932] T f69020bb6c244848a6b0a50fd2b4f3eb P 5bf81f3d986e428190d3f7727f36d96c [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:15:47.080981 6863 consensus_queue.cc:1035] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:15:47.081311 19233 catalog_manager.cc:5484] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d reported cstate change: config changed from index 10 to 11, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) evicted. New cstate: current_term: 2 leader_uuid: "43ce4a4c954f4a5f80954b38933c189d" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:15:47.089406 11815 raft_consensus.cc:2932] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:15:47.091415 21522 raft_consensus.cc:2932] T 410a7dd0dd734684a82875912966657d P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:15:47.091747 19227 catalog_manager.cc:5484] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc reported cstate change: config changed from index 9 to 10, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) evicted. New cstate: current_term: 1 leader_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
W20240501 14:15:47.098151 19219 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
W20240501 14:15:47.099874 19219 catalog_manager.cc:4573] TS e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet f69020bb6c244848a6b0a50fd2b4f3eb: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:47.103896 21399 consensus_queue.cc:237] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:15:47.105351 21137 raft_consensus.cc:1261] T f69020bb6c244848a6b0a50fd2b4f3eb P 5bf81f3d986e428190d3f7727f36d96c [term 2 FOLLOWER]: Refusing update from remote peer 43ce4a4c954f4a5f80954b38933c189d: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 14:15:47.105739 11812 consensus_queue.cc:1035] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:15:47.105759 19374 consensus_queue.cc:237] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:15:47.107021 21522 raft_consensus.cc:1261] T 410a7dd0dd734684a82875912966657d P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240501 14:15:47.107368 19311 consensus_peers.cc:485] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357): Couldn't send request to peer da3c4e948ff747678f18456c134d7ed4. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:15:47.107463 11815 consensus_queue.cc:1035] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:15:47.108026 11111 raft_consensus.cc:2932] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } }
I20240501 14:15:47.108543 21137 raft_consensus.cc:2932] T f69020bb6c244848a6b0a50fd2b4f3eb P 5bf81f3d986e428190d3f7727f36d96c [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } }
I20240501 14:15:47.110119 19218 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet f69020bb6c244848a6b0a50fd2b4f3eb with cas_config_opid_index 11: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:15:47.111063 19229 catalog_manager.cc:5484] T f69020bb6c244848a6b0a50fd2b4f3eb P 5bf81f3d986e428190d3f7727f36d96c reported cstate change: config changed from index 11 to 12, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) added. New cstate: current_term: 2 leader_uuid: "43ce4a4c954f4a5f80954b38933c189d" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } }
W20240501 14:15:47.111672 21332 consensus_peers.cc:485] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d -> Peer fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847): Couldn't send request to peer fb1e6bc2d07b4ce3b622a66f14c93bdc. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:15:47.113078 6863 raft_consensus.cc:2932] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } }
I20240501 14:15:47.115581 19227 catalog_manager.cc:5484] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc reported cstate change: config changed from index 10 to 11, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) added. New cstate: current_term: 1 leader_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } }
I20240501 14:15:47.115834 19218 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 410a7dd0dd734684a82875912966657d with cas_config_opid_index 10: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:15:47.116215 21522 raft_consensus.cc:2932] T 410a7dd0dd734684a82875912966657d P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } }
I20240501 14:15:47.175411 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:47.187155 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:47.187497 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:47.189361 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:47.194061 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:47.196223 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:47.204309 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:47.211297 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:47.211556 9990 raft_consensus.cc:998] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367: Attempting to remove follower e6246b183293412494919f1e7efc9448 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e6246b183293412494919f1e7efc9448 for more than 300 seconds (300.227s)
I20240501 14:15:47.211931 9990 consensus_queue.cc:237] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }
I20240501 14:15:47.214421 19374 raft_consensus.cc:1261] T ffb4b6a2c19b497f8117fbbc9ea50082 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:15:47.215268 9990 consensus_queue.cc:1035] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:15:47.227990 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:47.232926 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:47.235002 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:47.245625 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:47.253067 11913 raft_consensus.cc:2932] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } }
I20240501 14:15:47.255990 19233 catalog_manager.cc:5484] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 reported cstate change: config changed from index 5 to 6, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) evicted. New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } }
I20240501 14:15:47.260613 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:47.261265 19374 raft_consensus.cc:2932] T ffb4b6a2c19b497f8117fbbc9ea50082 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } }
I20240501 14:15:47.265362 19636 consensus_queue.cc:237] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:15:47.266188 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:47.268882 9990 raft_consensus.cc:998] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367: Attempting to remove follower e6246b183293412494919f1e7efc9448 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e6246b183293412494919f1e7efc9448 for more than 300 seconds (300.096s)
I20240501 14:15:47.269237 9990 consensus_queue.cc:237] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:15:47.270445 21399 raft_consensus.cc:1261] T 85593473500b4d5eaf9f8b0bf2296777 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:15:47.271162 11913 consensus_queue.cc:1035] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 14:15:47.272208 19573 consensus_peers.cc:485] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 -> Peer da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357): Couldn't send request to peer da3c4e948ff747678f18456c134d7ed4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ffb4b6a2c19b497f8117fbbc9ea50082. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:15:47.274274 19374 raft_consensus.cc:1261] T ffb4b6a2c19b497f8117fbbc9ea50082 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:15:47.276082 11913 raft_consensus.cc:998] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367: Attempting to remove follower e6246b183293412494919f1e7efc9448 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e6246b183293412494919f1e7efc9448 for more than 300 seconds (300.004s)
I20240501 14:15:47.276405 11913 consensus_queue.cc:237] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:15:47.276733 9990 consensus_queue.cc:1035] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:15:47.277686 19769 raft_consensus.cc:1261] T 661aaf83e3474966a257fcc039ef4784 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:15:47.281610 9990 consensus_queue.cc:1035] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:15:47.283215 9990 raft_consensus.cc:2932] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } }
I20240501 14:15:47.283272 19374 raft_consensus.cc:2932] T ffb4b6a2c19b497f8117fbbc9ea50082 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } }
I20240501 14:15:47.285899 19233 catalog_manager.cc:5484] T ffb4b6a2c19b497f8117fbbc9ea50082 P fb1e6bc2d07b4ce3b622a66f14c93bdc reported cstate change: config changed from index 6 to 7, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) added. New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } }
I20240501 14:15:47.295346 19219 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet ffb4b6a2c19b497f8117fbbc9ea50082 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:15:47.295512 11913 raft_consensus.cc:2932] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:15:47.295931 9990 raft_consensus.cc:2932] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } }
I20240501 14:15:47.296020 21399 raft_consensus.cc:2932] T 85593473500b4d5eaf9f8b0bf2296777 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:15:47.298002 19233 catalog_manager.cc:5484] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 reported cstate change: config changed from index 7 to 8, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) evicted. New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:15:47.298406 19769 raft_consensus.cc:2932] T 661aaf83e3474966a257fcc039ef4784 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } }
I20240501 14:15:47.301257 19230 catalog_manager.cc:5484] T 661aaf83e3474966a257fcc039ef4784 P da3c4e948ff747678f18456c134d7ed4 reported cstate change: config changed from index 5 to 6, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) evicted. New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } }
I20240501 14:15:47.301942 19636 consensus_queue.cc:237] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
W20240501 14:15:47.303601 19571 consensus_peers.cc:485] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 -> Peer fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847): Couldn't send request to peer fb1e6bc2d07b4ce3b622a66f14c93bdc. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:15:47.303721 21399 raft_consensus.cc:1261] T 85593473500b4d5eaf9f8b0bf2296777 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:15:47.305408 19636 consensus_queue.cc:237] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:15:47.305620 11921 consensus_queue.cc:1035] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 14:15:47.307265 19571 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136:34895): Couldn't send request to peer 778c78105092428bbcde75ab2cdb6a05. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 661aaf83e3474966a257fcc039ef4784. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:15:47.307520 19769 raft_consensus.cc:1261] T 661aaf83e3474966a257fcc039ef4784 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:15:47.307914 9990 consensus_queue.cc:1035] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:15:47.308833 11913 raft_consensus.cc:2932] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } }
I20240501 14:15:47.309250 21399 raft_consensus.cc:2932] T 85593473500b4d5eaf9f8b0bf2296777 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } }
I20240501 14:15:47.309830 9990 raft_consensus.cc:2932] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:15:47.312074 19219 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 85593473500b4d5eaf9f8b0bf2296777 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:15:47.312103 19769 raft_consensus.cc:2932] T 661aaf83e3474966a257fcc039ef4784 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:15:47.312244 19219 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 661aaf83e3474966a257fcc039ef4784 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:15:47.312079 19230 catalog_manager.cc:5484] T 85593473500b4d5eaf9f8b0bf2296777 P 43ce4a4c954f4a5f80954b38933c189d reported cstate change: config changed from index 8 to 9, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) added. New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } }
I20240501 14:15:47.315754 19233 catalog_manager.cc:5484] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 reported cstate change: config changed from index 6 to 7, VOTER 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136) added. New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:15:47.347733 11842 raft_consensus.cc:998] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Attempting to remove follower e6246b183293412494919f1e7efc9448 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e6246b183293412494919f1e7efc9448 for more than 300 seconds (300.015s)
I20240501 14:15:47.348026 11842 consensus_queue.cc:237] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:15:47.349073 21399 raft_consensus.cc:1261] T d63afd081986473aa2715d074b123a93 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:15:47.349437 11815 consensus_queue.cc:1035] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:15:47.353291 11842 raft_consensus.cc:2932] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:15:47.353673 21399 raft_consensus.cc:2932] T d63afd081986473aa2715d074b123a93 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:15:47.355487 19234 catalog_manager.cc:5484] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc reported cstate change: config changed from index 9 to 10, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) evicted. New cstate: current_term: 1 leader_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:15:47.359345 19374 consensus_queue.cc:237] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
W20240501 14:15:47.360579 19311 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357): Couldn't send request to peer da3c4e948ff747678f18456c134d7ed4. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:15:47.361091 21399 raft_consensus.cc:1261] T d63afd081986473aa2715d074b123a93 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:15:47.361469 11842 consensus_queue.cc:1035] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:15:47.362818 11815 raft_consensus.cc:2932] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } }
I20240501 14:15:47.364264 19218 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet d63afd081986473aa2715d074b123a93 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:15:47.364842 21399 raft_consensus.cc:2932] T d63afd081986473aa2715d074b123a93 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } }
I20240501 14:15:47.365065 19233 catalog_manager.cc:5484] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc reported cstate change: config changed from index 10 to 11, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) added. New cstate: current_term: 1 leader_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } }
I20240501 14:15:47.374984 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:47.379536 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:47.380437 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:47.383613 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:47.385859 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:47.393206 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:47.393782 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:47.484292 11916 raft_consensus.cc:998] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367: Attempting to remove follower e6246b183293412494919f1e7efc9448 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e6246b183293412494919f1e7efc9448 for more than 300 seconds (300.202s)
I20240501 14:15:47.484618 11916 consensus_queue.cc:237] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:15:47.485723 21399 raft_consensus.cc:1261] T cc8e888d46d34a22b6d4409ec2ab53a9 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:15:47.486102 11913 consensus_queue.cc:1035] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:15:47.487740 11916 raft_consensus.cc:2932] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:15:47.488102 21399 raft_consensus.cc:2932] T cc8e888d46d34a22b6d4409ec2ab53a9 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:15:47.490965 19228 catalog_manager.cc:5484] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 reported cstate change: config changed from index 7 to 8, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) evicted. New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:15:47.493772 19636 consensus_queue.cc:237] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:15:47.494858 21399 raft_consensus.cc:1261] T cc8e888d46d34a22b6d4409ec2ab53a9 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240501 14:15:47.495101 19573 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357): Couldn't send request to peer da3c4e948ff747678f18456c134d7ed4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cc8e888d46d34a22b6d4409ec2ab53a9. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:15:47.495211 11913 consensus_queue.cc:1035] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:15:47.496592 11916 raft_consensus.cc:2932] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } }
I20240501 14:15:47.497037 21399 raft_consensus.cc:2932] T cc8e888d46d34a22b6d4409ec2ab53a9 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } }
I20240501 14:15:47.498834 19233 catalog_manager.cc:5484] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 reported cstate change: config changed from index 8 to 9, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) added. New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } }
I20240501 14:15:47.498831 19219 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet cc8e888d46d34a22b6d4409ec2ab53a9 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:15:47.505647 11913 raft_consensus.cc:998] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367: Attempting to remove follower e6246b183293412494919f1e7efc9448 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e6246b183293412494919f1e7efc9448 for more than 300 seconds (300.100s)
I20240501 14:15:47.505964 11913 consensus_queue.cc:237] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:15:47.507155 19769 raft_consensus.cc:1261] T db0f01ee72b24ec9b84410cf94ef9759 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:15:47.507223 21399 raft_consensus.cc:1261] T db0f01ee72b24ec9b84410cf94ef9759 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:15:47.507566 11921 consensus_queue.cc:1035] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:15:47.508101 11921 consensus_queue.cc:1035] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:15:47.509852 11927 raft_consensus.cc:2932] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } }
I20240501 14:15:47.510275 21399 raft_consensus.cc:2932] T db0f01ee72b24ec9b84410cf94ef9759 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } }
I20240501 14:15:47.510586 19769 raft_consensus.cc:2932] T db0f01ee72b24ec9b84410cf94ef9759 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } }
I20240501 14:15:47.512817 19228 catalog_manager.cc:5484] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 reported cstate change: config changed from index 8 to 9, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) evicted. New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } }
I20240501 14:15:47.529937 11927 raft_consensus.cc:998] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367: Attempting to remove follower e6246b183293412494919f1e7efc9448 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e6246b183293412494919f1e7efc9448 for more than 300 seconds (300.456s)
I20240501 14:15:47.530249 11927 consensus_queue.cc:237] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:15:47.531425 21522 raft_consensus.cc:1261] T fe554c3145de46cf8070efcecffbaedb P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:15:47.531900 11921 consensus_queue.cc:1035] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:15:47.535122 11921 raft_consensus.cc:2932] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:15:47.538012 21522 raft_consensus.cc:2932] T fe554c3145de46cf8070efcecffbaedb P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:15:47.538383 19225 catalog_manager.cc:5484] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 reported cstate change: config changed from index 7 to 8, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) evicted. New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:15:47.542107 19636 consensus_queue.cc:237] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:15:47.544454 11913 raft_consensus.cc:998] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367: Attempting to remove follower e6246b183293412494919f1e7efc9448 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e6246b183293412494919f1e7efc9448 for more than 300 seconds (300.214s)
I20240501 14:15:47.544765 11913 consensus_queue.cc:237] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:15:47.546418 21522 raft_consensus.cc:1261] T fe554c3145de46cf8070efcecffbaedb P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240501 14:15:47.546525 19571 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135:38751): Couldn't send request to peer 43ce4a4c954f4a5f80954b38933c189d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fe554c3145de46cf8070efcecffbaedb. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:15:47.547181 21399 raft_consensus.cc:1261] T 2dcca6e0e6374520bd4522c80c110e97 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:15:47.548182 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:47.549459 11916 consensus_queue.cc:1035] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:15:47.549805 11916 consensus_queue.cc:1035] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:15:47.550843 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:47.555835 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:47.557130 11916 raft_consensus.cc:2932] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:15:47.558985 19219 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet fe554c3145de46cf8070efcecffbaedb with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:15:47.560500 21522 raft_consensus.cc:2932] T fe554c3145de46cf8070efcecffbaedb P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:15:47.560743 19225 catalog_manager.cc:5484] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 reported cstate change: config changed from index 8 to 9, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:15:47.565125 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:47.566473 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:47.566612 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:47.574476 11921 raft_consensus.cc:2932] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:15:47.576828 21399 raft_consensus.cc:2932] T 2dcca6e0e6374520bd4522c80c110e97 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:15:47.577136 19226 catalog_manager.cc:5484] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 reported cstate change: config changed from index 7 to 8, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) evicted. New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:15:47.580902 19636 consensus_queue.cc:237] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:15:47.582250 21399 raft_consensus.cc:1261] T 2dcca6e0e6374520bd4522c80c110e97 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:15:47.582635 11913 consensus_queue.cc:1035] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 14:15:47.585074 19571 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847): Couldn't send request to peer fb1e6bc2d07b4ce3b622a66f14c93bdc. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:15:47.586374 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:47.586994 12016 ts_tablet_manager.cc:902] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4: Initiating tablet copy from peer fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847)
I20240501 14:15:47.587604 12016 tablet_copy_client.cc:308] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.129:35847
I20240501 14:15:47.588081 19396 tablet_copy_service.cc:140] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Received BeginTabletCopySession request for tablet 410a7dd0dd734684a82875912966657d from peer da3c4e948ff747678f18456c134d7ed4 ({username='slave'} at 127.7.39.132:46133)
I20240501 14:15:47.588207 19396 tablet_copy_service.cc:161] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Beginning new tablet copy session on tablet 410a7dd0dd734684a82875912966657d from peer da3c4e948ff747678f18456c134d7ed4 at {username='slave'} at 127.7.39.132:46133: session id = da3c4e948ff747678f18456c134d7ed4-410a7dd0dd734684a82875912966657d
I20240501 14:15:47.589591 19396 tablet_copy_source_session.cc:205] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 14:15:47.590536 12016 ts_tablet_manager.cc:1875] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:15:47.593008 12016 ts_tablet_manager.cc:1888] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.7
I20240501 14:15:47.593283 12016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 410a7dd0dd734684a82875912966657d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:15:47.593390 11913 raft_consensus.cc:2932] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } }
I20240501 14:15:47.594732 12016 tablet_copy_client.cc:783] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4: tablet copy: Starting download of 5 data blocks...
I20240501 14:15:47.595472 19219 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 2dcca6e0e6374520bd4522c80c110e97 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:15:47.595911 21399 raft_consensus.cc:2932] T 2dcca6e0e6374520bd4522c80c110e97 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } }
I20240501 14:15:47.596288 19226 catalog_manager.cc:5484] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 reported cstate change: config changed from index 8 to 9, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) added. New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } }
I20240501 14:15:47.606034 12016 tablet_copy_client.cc:647] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4: tablet copy: Starting download of 1 WAL segments...
I20240501 14:15:47.607502 12016 tablet_copy_client.cc:523] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:15:47.609184 12016 tablet_bootstrap.cc:492] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4: Bootstrap starting.
I20240501 14:15:47.618489 11922 raft_consensus.cc:998] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4: Attempting to remove follower e6246b183293412494919f1e7efc9448 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e6246b183293412494919f1e7efc9448 for more than 300 seconds (300.287s)
I20240501 14:15:47.618800 11922 consensus_queue.cc:237] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:15:47.619590 12016 tablet_bootstrap.cc:492] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:15:47.619894 12016 tablet_bootstrap.cc:492] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4: Bootstrap complete.
I20240501 14:15:47.620077 12016 ts_tablet_manager.cc:1367] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.001s
I20240501 14:15:47.620076 21137 raft_consensus.cc:1261] T 11f64b59f83e458f8cf51fd5bcd36fee P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:15:47.620443 12016 raft_consensus.cc:348] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:15:47.620545 12016 raft_consensus.cc:729] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3c4e948ff747678f18456c134d7ed4, State: Initialized, Role: FOLLOWER
I20240501 14:15:47.620730 12016 consensus_queue.cc:260] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:15:47.621080 12016 ts_tablet_manager.cc:1398] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:15:47.621357 11026 consensus_queue.cc:1035] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:15:47.621639 19396 tablet_copy_service.cc:342] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Request end of tablet copy session da3c4e948ff747678f18456c134d7ed4-410a7dd0dd734684a82875912966657d received from {username='slave'} at 127.7.39.132:46133
I20240501 14:15:47.621722 19396 tablet_copy_service.cc:433] P fb1e6bc2d07b4ce3b622a66f14c93bdc: ending tablet copy session da3c4e948ff747678f18456c134d7ed4-410a7dd0dd734684a82875912966657d on tablet 410a7dd0dd734684a82875912966657d with peer da3c4e948ff747678f18456c134d7ed4
I20240501 14:15:47.624825 11922 raft_consensus.cc:2932] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:15:47.626293 21137 raft_consensus.cc:2932] T 11f64b59f83e458f8cf51fd5bcd36fee P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:15:47.627579 19225 catalog_manager.cc:5484] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 reported cstate change: config changed from index 7 to 8, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) evicted. New cstate: current_term: 1 leader_uuid: "da3c4e948ff747678f18456c134d7ed4" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:15:47.634904 19769 consensus_queue.cc:237] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:15:47.636410 21137 raft_consensus.cc:1261] T 11f64b59f83e458f8cf51fd5bcd36fee P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:15:47.636813 11922 consensus_queue.cc:1035] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 14:15:47.637262 19703 consensus_peers.cc:485] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 -> Peer 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829): Couldn't send request to peer 99a0be6ef2144b7fb247ba9d3f051d66. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 11f64b59f83e458f8cf51fd5bcd36fee. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:15:47.642804 11922 raft_consensus.cc:2932] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } }
I20240501 14:15:47.643472 21137 raft_consensus.cc:2932] T 11f64b59f83e458f8cf51fd5bcd36fee P 5bf81f3d986e428190d3f7727f36d96c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } }
I20240501 14:15:47.645993 19225 catalog_manager.cc:5484] T 11f64b59f83e458f8cf51fd5bcd36fee P 5bf81f3d986e428190d3f7727f36d96c reported cstate change: config changed from index 8 to 9, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) added. New cstate: current_term: 1 leader_uuid: "da3c4e948ff747678f18456c134d7ed4" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } }
I20240501 14:15:47.646972 19220 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 11f64b59f83e458f8cf51fd5bcd36fee with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:15:47.661299 12016 ts_tablet_manager.cc:902] T ffb4b6a2c19b497f8117fbbc9ea50082 P da3c4e948ff747678f18456c134d7ed4: Initiating tablet copy from peer ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201)
I20240501 14:15:47.661739 12016 tablet_copy_client.cc:308] T ffb4b6a2c19b497f8117fbbc9ea50082 P da3c4e948ff747678f18456c134d7ed4: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.131:45201
I20240501 14:15:47.666155 19657 tablet_copy_service.cc:140] P ec8cde17a00645e7b38d5d29f7425367: Received BeginTabletCopySession request for tablet ffb4b6a2c19b497f8117fbbc9ea50082 from peer da3c4e948ff747678f18456c134d7ed4 ({username='slave'} at 127.7.39.132:34227)
I20240501 14:15:47.666257 19657 tablet_copy_service.cc:161] P ec8cde17a00645e7b38d5d29f7425367: Beginning new tablet copy session on tablet ffb4b6a2c19b497f8117fbbc9ea50082 from peer da3c4e948ff747678f18456c134d7ed4 at {username='slave'} at 127.7.39.132:34227: session id = da3c4e948ff747678f18456c134d7ed4-ffb4b6a2c19b497f8117fbbc9ea50082
I20240501 14:15:47.667340 19657 tablet_copy_source_session.cc:205] T ffb4b6a2c19b497f8117fbbc9ea50082 P ec8cde17a00645e7b38d5d29f7425367: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 14:15:47.668262 12016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffb4b6a2c19b497f8117fbbc9ea50082. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:15:47.670346 12016 tablet_copy_client.cc:783] T ffb4b6a2c19b497f8117fbbc9ea50082 P da3c4e948ff747678f18456c134d7ed4: tablet copy: Starting download of 5 data blocks...
I20240501 14:15:47.677601 12016 tablet_copy_client.cc:647] T ffb4b6a2c19b497f8117fbbc9ea50082 P da3c4e948ff747678f18456c134d7ed4: tablet copy: Starting download of 1 WAL segments...
I20240501 14:15:47.678886 12016 tablet_copy_client.cc:523] T ffb4b6a2c19b497f8117fbbc9ea50082 P da3c4e948ff747678f18456c134d7ed4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:15:47.680091 12016 tablet_bootstrap.cc:492] T ffb4b6a2c19b497f8117fbbc9ea50082 P da3c4e948ff747678f18456c134d7ed4: Bootstrap starting.
I20240501 14:15:47.698415 12016 tablet_bootstrap.cc:492] T ffb4b6a2c19b497f8117fbbc9ea50082 P da3c4e948ff747678f18456c134d7ed4: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:15:47.698746 12016 tablet_bootstrap.cc:492] T ffb4b6a2c19b497f8117fbbc9ea50082 P da3c4e948ff747678f18456c134d7ed4: Bootstrap complete.
I20240501 14:15:47.698949 12016 ts_tablet_manager.cc:1367] T ffb4b6a2c19b497f8117fbbc9ea50082 P da3c4e948ff747678f18456c134d7ed4: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.000s
I20240501 14:15:47.699431 12016 raft_consensus.cc:348] T ffb4b6a2c19b497f8117fbbc9ea50082 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:15:47.699604 12016 raft_consensus.cc:729] T ffb4b6a2c19b497f8117fbbc9ea50082 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3c4e948ff747678f18456c134d7ed4, State: Initialized, Role: FOLLOWER
I20240501 14:15:47.699796 12016 consensus_queue.cc:260] T ffb4b6a2c19b497f8117fbbc9ea50082 P da3c4e948ff747678f18456c134d7ed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:15:47.702996 12016 ts_tablet_manager.cc:1398] T ffb4b6a2c19b497f8117fbbc9ea50082 P da3c4e948ff747678f18456c134d7ed4: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:15:47.703850 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:47.704563 19657 tablet_copy_service.cc:342] P ec8cde17a00645e7b38d5d29f7425367: Request end of tablet copy session da3c4e948ff747678f18456c134d7ed4-ffb4b6a2c19b497f8117fbbc9ea50082 received from {username='slave'} at 127.7.39.132:34227
I20240501 14:15:47.704645 19657 tablet_copy_service.cc:433] P ec8cde17a00645e7b38d5d29f7425367: ending tablet copy session da3c4e948ff747678f18456c134d7ed4-ffb4b6a2c19b497f8117fbbc9ea50082 on tablet ffb4b6a2c19b497f8117fbbc9ea50082 with peer da3c4e948ff747678f18456c134d7ed4
I20240501 14:15:47.704885 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:47.724618 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:47.729179 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:47.729331 12114 ts_tablet_manager.cc:902] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc: Initiating tablet copy from peer 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135:38751)
I20240501 14:15:47.731398 12114 tablet_copy_client.cc:308] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.135:38751
I20240501 14:15:47.732118 11965 raft_consensus.cc:998] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66: Attempting to remove follower e6246b183293412494919f1e7efc9448 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e6246b183293412494919f1e7efc9448 for more than 300 seconds (300.491s)
I20240501 14:15:47.732537 11965 consensus_queue.cc:237] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:15:47.732793 21419 tablet_copy_service.cc:140] P 43ce4a4c954f4a5f80954b38933c189d: Received BeginTabletCopySession request for tablet f69020bb6c244848a6b0a50fd2b4f3eb from peer fb1e6bc2d07b4ce3b622a66f14c93bdc ({username='slave'} at 127.7.39.129:58141)
I20240501 14:15:47.732875 21419 tablet_copy_service.cc:161] P 43ce4a4c954f4a5f80954b38933c189d: Beginning new tablet copy session on tablet f69020bb6c244848a6b0a50fd2b4f3eb from peer fb1e6bc2d07b4ce3b622a66f14c93bdc at {username='slave'} at 127.7.39.129:58141: session id = fb1e6bc2d07b4ce3b622a66f14c93bdc-f69020bb6c244848a6b0a50fd2b4f3eb
I20240501 14:15:47.734265 21419 tablet_copy_source_session.cc:205] T f69020bb6c244848a6b0a50fd2b4f3eb P 43ce4a4c954f4a5f80954b38933c189d: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 14:15:47.735446 12114 ts_tablet_manager.cc:1875] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:15:47.738660 12114 ts_tablet_manager.cc:1888] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.8
I20240501 14:15:47.738890 12114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f69020bb6c244848a6b0a50fd2b4f3eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:15:47.740101 12114 tablet_copy_client.cc:783] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc: tablet copy: Starting download of 5 data blocks...
I20240501 14:15:47.740320 11842 raft_consensus.cc:998] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc: Attempting to remove follower e6246b183293412494919f1e7efc9448 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e6246b183293412494919f1e7efc9448 for more than 300 seconds (300.425s)
I20240501 14:15:47.741505 11842 consensus_queue.cc:237] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:15:47.742635 21399 raft_consensus.cc:1261] T ad7c255baee74e629e2976a942a4174f P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:15:47.743192 11815 consensus_queue.cc:1035] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:15:47.743950 12127 ts_tablet_manager.cc:902] T 661aaf83e3474966a257fcc039ef4784 P 778c78105092428bbcde75ab2cdb6a05: Initiating tablet copy from peer ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201)
I20240501 14:15:47.744722 12127 tablet_copy_client.cc:308] T 661aaf83e3474966a257fcc039ef4784 P 778c78105092428bbcde75ab2cdb6a05: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.131:45201
I20240501 14:15:47.745662 11815 raft_consensus.cc:2932] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:15:47.747277 21522 raft_consensus.cc:1261] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Refusing update from remote peer 99a0be6ef2144b7fb247ba9d3f051d66: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:15:47.747581 11965 consensus_queue.cc:1035] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:15:47.748512 21399 raft_consensus.cc:2932] T ad7c255baee74e629e2976a942a4174f P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:15:47.749147 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:47.749742 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:47.753515 19225 catalog_manager.cc:5484] T ad7c255baee74e629e2976a942a4174f P 43ce4a4c954f4a5f80954b38933c189d reported cstate change: config changed from index 9 to 10, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) evicted. New cstate: current_term: 1 leader_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:15:47.754694 12114 tablet_copy_client.cc:647] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc: tablet copy: Starting download of 1 WAL segments...
I20240501 14:15:47.755887 12114 tablet_copy_client.cc:523] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:15:47.757597 12114 tablet_bootstrap.cc:492] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc: Bootstrap starting.
I20240501 14:15:47.763387 19374 consensus_queue.cc:237] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:15:47.764310 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:15:47.764935 19310 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829): Couldn't send request to peer 99a0be6ef2144b7fb247ba9d3f051d66. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:15:47.765005 19657 tablet_copy_service.cc:140] P ec8cde17a00645e7b38d5d29f7425367: Received BeginTabletCopySession request for tablet 661aaf83e3474966a257fcc039ef4784 from peer 778c78105092428bbcde75ab2cdb6a05 ({username='slave'} at 127.7.39.136:60681)
I20240501 14:15:47.765091 19657 tablet_copy_service.cc:161] P ec8cde17a00645e7b38d5d29f7425367: Beginning new tablet copy session on tablet 661aaf83e3474966a257fcc039ef4784 from peer 778c78105092428bbcde75ab2cdb6a05 at {username='slave'} at 127.7.39.136:60681: session id = 778c78105092428bbcde75ab2cdb6a05-661aaf83e3474966a257fcc039ef4784
I20240501 14:15:47.765551 21398 raft_consensus.cc:1261] T ad7c255baee74e629e2976a942a4174f P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Refusing update from remote peer fb1e6bc2d07b4ce3b622a66f14c93bdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:15:47.765898 6863 consensus_queue.cc:1035] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:15:47.766014 19657 tablet_copy_source_session.cc:205] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 14:15:47.767237 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:47.767863 6863 raft_consensus.cc:2932] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } }
I20240501 14:15:47.769490 21398 raft_consensus.cc:2932] T ad7c255baee74e629e2976a942a4174f P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } }
I20240501 14:15:47.771173 19218 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet ad7c255baee74e629e2976a942a4174f with cas_config_opid_index 10: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:15:47.771417 12127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 661aaf83e3474966a257fcc039ef4784. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:15:47.773362 19225 catalog_manager.cc:5484] T ad7c255baee74e629e2976a942a4174f P 43ce4a4c954f4a5f80954b38933c189d reported cstate change: config changed from index 10 to 11, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) added. New cstate: current_term: 1 leader_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } }
I20240501 14:15:47.773919 12127 tablet_copy_client.cc:783] T 661aaf83e3474966a257fcc039ef4784 P 778c78105092428bbcde75ab2cdb6a05: tablet copy: Starting download of 5 data blocks...
I20240501 14:15:47.774902 12114 tablet_bootstrap.cc:492] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:15:47.775178 12114 tablet_bootstrap.cc:492] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc: Bootstrap complete.
I20240501 14:15:47.775363 12114 ts_tablet_manager.cc:1367] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.000s
I20240501 14:15:47.775653 12114 raft_consensus.cc:348] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:15:47.775715 12114 raft_consensus.cc:729] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1e6bc2d07b4ce3b622a66f14c93bdc, State: Initialized, Role: FOLLOWER
I20240501 14:15:47.775804 12114 consensus_queue.cc:260] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:15:47.776023 12114 ts_tablet_manager.cc:1398] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:15:47.776453 21419 tablet_copy_service.cc:342] P 43ce4a4c954f4a5f80954b38933c189d: Request end of tablet copy session fb1e6bc2d07b4ce3b622a66f14c93bdc-f69020bb6c244848a6b0a50fd2b4f3eb received from {username='slave'} at 127.7.39.129:58141
I20240501 14:15:47.776512 21419 tablet_copy_service.cc:433] P 43ce4a4c954f4a5f80954b38933c189d: ending tablet copy session fb1e6bc2d07b4ce3b622a66f14c93bdc-f69020bb6c244848a6b0a50fd2b4f3eb on tablet f69020bb6c244848a6b0a50fd2b4f3eb with peer fb1e6bc2d07b4ce3b622a66f14c93bdc
I20240501 14:15:47.777479 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:47.778195 19423 maintenance_manager.cc:419] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Scheduling UndoDeltaBlockGCOp(f69020bb6c244848a6b0a50fd2b4f3eb): 4921 bytes on disk
I20240501 14:15:47.778527 19313 maintenance_manager.cc:643] P fb1e6bc2d07b4ce3b622a66f14c93bdc: UndoDeltaBlockGCOp(f69020bb6c244848a6b0a50fd2b4f3eb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4}
I20240501 14:15:47.780514 12114 ts_tablet_manager.cc:902] T 85593473500b4d5eaf9f8b0bf2296777 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Initiating tablet copy from peer ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201)
I20240501 14:15:47.780891 12114 tablet_copy_client.cc:308] T 85593473500b4d5eaf9f8b0bf2296777 P fb1e6bc2d07b4ce3b622a66f14c93bdc: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.131:45201
I20240501 14:15:47.790488 19657 tablet_copy_service.cc:140] P ec8cde17a00645e7b38d5d29f7425367: Received BeginTabletCopySession request for tablet 85593473500b4d5eaf9f8b0bf2296777 from peer fb1e6bc2d07b4ce3b622a66f14c93bdc ({username='slave'} at 127.7.39.129:47463)
I20240501 14:15:47.790570 19657 tablet_copy_service.cc:161] P ec8cde17a00645e7b38d5d29f7425367: Beginning new tablet copy session on tablet 85593473500b4d5eaf9f8b0bf2296777 from peer fb1e6bc2d07b4ce3b622a66f14c93bdc at {username='slave'} at 127.7.39.129:47463: session id = fb1e6bc2d07b4ce3b622a66f14c93bdc-85593473500b4d5eaf9f8b0bf2296777
I20240501 14:15:47.791353 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:47.791484 19657 tablet_copy_source_session.cc:205] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 14:15:47.795831 12114 ts_tablet_manager.cc:1875] T 85593473500b4d5eaf9f8b0bf2296777 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:15:47.795847 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:47.797950 12114 ts_tablet_manager.cc:1888] T 85593473500b4d5eaf9f8b0bf2296777 P fb1e6bc2d07b4ce3b622a66f14c93bdc: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 14:15:47.798096 12114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85593473500b4d5eaf9f8b0bf2296777. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:15:47.799132 12114 tablet_copy_client.cc:783] T 85593473500b4d5eaf9f8b0bf2296777 P fb1e6bc2d07b4ce3b622a66f14c93bdc: tablet copy: Starting download of 5 data blocks...
I20240501 14:15:47.808235 12016 ts_tablet_manager.cc:902] T d63afd081986473aa2715d074b123a93 P da3c4e948ff747678f18456c134d7ed4: Initiating tablet copy from peer fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847)
I20240501 14:15:47.808652 12016 tablet_copy_client.cc:308] T d63afd081986473aa2715d074b123a93 P da3c4e948ff747678f18456c134d7ed4: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.129:35847
I20240501 14:15:47.808995 19396 tablet_copy_service.cc:140] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Received BeginTabletCopySession request for tablet d63afd081986473aa2715d074b123a93 from peer da3c4e948ff747678f18456c134d7ed4 ({username='slave'} at 127.7.39.132:46133)
I20240501 14:15:47.809070 19396 tablet_copy_service.cc:161] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Beginning new tablet copy session on tablet d63afd081986473aa2715d074b123a93 from peer da3c4e948ff747678f18456c134d7ed4 at {username='slave'} at 127.7.39.132:46133: session id = da3c4e948ff747678f18456c134d7ed4-d63afd081986473aa2715d074b123a93
I20240501 14:15:47.809974 19396 tablet_copy_source_session.cc:205] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 14:15:47.811259 12016 ts_tablet_manager.cc:1875] T d63afd081986473aa2715d074b123a93 P da3c4e948ff747678f18456c134d7ed4: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:15:47.817365 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:47.819252 12127 tablet_copy_client.cc:647] T 661aaf83e3474966a257fcc039ef4784 P 778c78105092428bbcde75ab2cdb6a05: tablet copy: Starting download of 1 WAL segments...
I20240501 14:15:47.820721 12127 tablet_copy_client.cc:523] T 661aaf83e3474966a257fcc039ef4784 P 778c78105092428bbcde75ab2cdb6a05: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:15:47.822170 12127 tablet_bootstrap.cc:492] T 661aaf83e3474966a257fcc039ef4784 P 778c78105092428bbcde75ab2cdb6a05: Bootstrap starting.
I20240501 14:15:47.826779 12114 tablet_copy_client.cc:647] T 85593473500b4d5eaf9f8b0bf2296777 P fb1e6bc2d07b4ce3b622a66f14c93bdc: tablet copy: Starting download of 1 WAL segments...
I20240501 14:15:47.828110 12114 tablet_copy_client.cc:523] T 85593473500b4d5eaf9f8b0bf2296777 P fb1e6bc2d07b4ce3b622a66f14c93bdc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:15:47.829116 12016 ts_tablet_manager.cc:1888] T d63afd081986473aa2715d074b123a93 P da3c4e948ff747678f18456c134d7ed4: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 14:15:47.829421 12114 tablet_bootstrap.cc:492] T 85593473500b4d5eaf9f8b0bf2296777 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Bootstrap starting.
I20240501 14:15:47.830981 12016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d63afd081986473aa2715d074b123a93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:15:47.835485 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:47.835681 11965 raft_consensus.cc:2932] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:15:47.838586 21522 raft_consensus.cc:2932] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:15:47.838924 19234 catalog_manager.cc:5484] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 reported cstate change: config changed from index 7 to 8, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) evicted. New cstate: current_term: 1 leader_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:15:47.839318 19816 maintenance_manager.cc:419] P da3c4e948ff747678f18456c134d7ed4: Scheduling UndoDeltaBlockGCOp(ffb4b6a2c19b497f8117fbbc9ea50082): 4921 bytes on disk
I20240501 14:15:47.839725 19706 maintenance_manager.cc:643] P da3c4e948ff747678f18456c134d7ed4: UndoDeltaBlockGCOp(ffb4b6a2c19b497f8117fbbc9ea50082) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
I20240501 14:15:47.840360 19816 maintenance_manager.cc:419] P da3c4e948ff747678f18456c134d7ed4: Scheduling UndoDeltaBlockGCOp(410a7dd0dd734684a82875912966657d): 4832 bytes on disk
I20240501 14:15:47.840739 19706 maintenance_manager.cc:643] P da3c4e948ff747678f18456c134d7ed4: UndoDeltaBlockGCOp(410a7dd0dd734684a82875912966657d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
I20240501 14:15:47.849501 19506 consensus_queue.cc:237] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:15:47.850930 21522 raft_consensus.cc:1261] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Refusing update from remote peer 99a0be6ef2144b7fb247ba9d3f051d66: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:15:47.852039 11563 consensus_queue.cc:1035] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:15:47.852110 12114 tablet_bootstrap.cc:492] T 85593473500b4d5eaf9f8b0bf2296777 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:15:47.852584 12114 tablet_bootstrap.cc:492] T 85593473500b4d5eaf9f8b0bf2296777 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Bootstrap complete.
I20240501 14:15:47.852989 12114 ts_tablet_manager.cc:1367] T 85593473500b4d5eaf9f8b0bf2296777 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.000s
I20240501 14:15:47.853489 12114 raft_consensus.cc:348] T 85593473500b4d5eaf9f8b0bf2296777 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:15:47.853600 12114 raft_consensus.cc:729] T 85593473500b4d5eaf9f8b0bf2296777 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1e6bc2d07b4ce3b622a66f14c93bdc, State: Initialized, Role: FOLLOWER
I20240501 14:15:47.853806 12114 consensus_queue.cc:260] T 85593473500b4d5eaf9f8b0bf2296777 P fb1e6bc2d07b4ce3b622a66f14c93bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:15:47.854089 12114 ts_tablet_manager.cc:1398] T 85593473500b4d5eaf9f8b0bf2296777 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:15:47.856323 19657 tablet_copy_service.cc:342] P ec8cde17a00645e7b38d5d29f7425367: Request end of tablet copy session fb1e6bc2d07b4ce3b622a66f14c93bdc-85593473500b4d5eaf9f8b0bf2296777 received from {username='slave'} at 127.7.39.129:47463
I20240501 14:15:47.856408 19657 tablet_copy_service.cc:433] P ec8cde17a00645e7b38d5d29f7425367: ending tablet copy session fb1e6bc2d07b4ce3b622a66f14c93bdc-85593473500b4d5eaf9f8b0bf2296777 on tablet 85593473500b4d5eaf9f8b0bf2296777 with peer fb1e6bc2d07b4ce3b622a66f14c93bdc
W20240501 14:15:47.857085 19440 consensus_peers.cc:485] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847): Couldn't send request to peer fb1e6bc2d07b4ce3b622a66f14c93bdc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 272eee3450cf4f48a4e1de2dcb28c0f7. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:15:47.858508 12150 raft_consensus.cc:2932] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } }
I20240501 14:15:47.859602 12016 tablet_copy_client.cc:783] T d63afd081986473aa2715d074b123a93 P da3c4e948ff747678f18456c134d7ed4: tablet copy: Starting download of 5 data blocks...
I20240501 14:15:47.860248 19219 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 272eee3450cf4f48a4e1de2dcb28c0f7 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:15:47.861157 21522 raft_consensus.cc:2932] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } }
I20240501 14:15:47.863366 19225 catalog_manager.cc:5484] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66 reported cstate change: config changed from index 8 to 9, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) added. New cstate: current_term: 1 leader_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } }
I20240501 14:15:47.870178 12127 tablet_bootstrap.cc:492] T 661aaf83e3474966a257fcc039ef4784 P 778c78105092428bbcde75ab2cdb6a05: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:15:47.870435 12127 tablet_bootstrap.cc:492] T 661aaf83e3474966a257fcc039ef4784 P 778c78105092428bbcde75ab2cdb6a05: Bootstrap complete.
I20240501 14:15:47.870585 12127 ts_tablet_manager.cc:1367] T 661aaf83e3474966a257fcc039ef4784 P 778c78105092428bbcde75ab2cdb6a05: Time spent bootstrapping tablet: real 0.048s user 0.007s sys 0.000s
I20240501 14:15:47.870944 12127 raft_consensus.cc:348] T 661aaf83e3474966a257fcc039ef4784 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:15:47.871021 12127 raft_consensus.cc:729] T 661aaf83e3474966a257fcc039ef4784 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 778c78105092428bbcde75ab2cdb6a05, State: Initialized, Role: FOLLOWER
I20240501 14:15:47.871181 12016 tablet_copy_client.cc:647] T d63afd081986473aa2715d074b123a93 P da3c4e948ff747678f18456c134d7ed4: tablet copy: Starting download of 1 WAL segments...
I20240501 14:15:47.872488 12016 tablet_copy_client.cc:523] T d63afd081986473aa2715d074b123a93 P da3c4e948ff747678f18456c134d7ed4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:15:47.876685 12016 tablet_bootstrap.cc:492] T d63afd081986473aa2715d074b123a93 P da3c4e948ff747678f18456c134d7ed4: Bootstrap starting.
I20240501 14:15:47.879387 12127 consensus_queue.cc:260] T 661aaf83e3474966a257fcc039ef4784 P 778c78105092428bbcde75ab2cdb6a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:15:47.880182 12127 ts_tablet_manager.cc:1398] T 661aaf83e3474966a257fcc039ef4784 P 778c78105092428bbcde75ab2cdb6a05: Time spent starting tablet: real 0.010s user 0.000s sys 0.004s
I20240501 14:15:47.881152 19657 tablet_copy_service.cc:342] P ec8cde17a00645e7b38d5d29f7425367: Request end of tablet copy session 778c78105092428bbcde75ab2cdb6a05-661aaf83e3474966a257fcc039ef4784 received from {username='slave'} at 127.7.39.136:60681
I20240501 14:15:47.881229 19657 tablet_copy_service.cc:433] P ec8cde17a00645e7b38d5d29f7425367: ending tablet copy session 778c78105092428bbcde75ab2cdb6a05-661aaf83e3474966a257fcc039ef4784 on tablet 661aaf83e3474966a257fcc039ef4784 with peer 778c78105092428bbcde75ab2cdb6a05
I20240501 14:15:47.882974 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:47.885368 12016 tablet_bootstrap.cc:492] T d63afd081986473aa2715d074b123a93 P da3c4e948ff747678f18456c134d7ed4: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:15:47.885601 12016 tablet_bootstrap.cc:492] T d63afd081986473aa2715d074b123a93 P da3c4e948ff747678f18456c134d7ed4: Bootstrap complete.
I20240501 14:15:47.885730 12016 ts_tablet_manager.cc:1367] T d63afd081986473aa2715d074b123a93 P da3c4e948ff747678f18456c134d7ed4: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20240501 14:15:47.886087 12016 raft_consensus.cc:348] T d63afd081986473aa2715d074b123a93 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:15:47.886196 12016 raft_consensus.cc:729] T d63afd081986473aa2715d074b123a93 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3c4e948ff747678f18456c134d7ed4, State: Initialized, Role: FOLLOWER
I20240501 14:15:47.886323 12016 consensus_queue.cc:260] T d63afd081986473aa2715d074b123a93 P da3c4e948ff747678f18456c134d7ed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:15:47.887017 12016 ts_tablet_manager.cc:1398] T d63afd081986473aa2715d074b123a93 P da3c4e948ff747678f18456c134d7ed4: Time spent starting tablet: real 0.001s user 0.003s sys 0.001s
I20240501 14:15:47.887595 19396 tablet_copy_service.cc:342] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Request end of tablet copy session da3c4e948ff747678f18456c134d7ed4-d63afd081986473aa2715d074b123a93 received from {username='slave'} at 127.7.39.132:46133
I20240501 14:15:47.887661 19396 tablet_copy_service.cc:433] P fb1e6bc2d07b4ce3b622a66f14c93bdc: ending tablet copy session da3c4e948ff747678f18456c134d7ed4-d63afd081986473aa2715d074b123a93 on tablet d63afd081986473aa2715d074b123a93 with peer da3c4e948ff747678f18456c134d7ed4
I20240501 14:15:47.888729 12016 ts_tablet_manager.cc:902] T cc8e888d46d34a22b6d4409ec2ab53a9 P da3c4e948ff747678f18456c134d7ed4: Initiating tablet copy from peer ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201)
I20240501 14:15:47.889180 12016 tablet_copy_client.cc:308] T cc8e888d46d34a22b6d4409ec2ab53a9 P da3c4e948ff747678f18456c134d7ed4: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.131:45201
I20240501 14:15:47.889633 19657 tablet_copy_service.cc:140] P ec8cde17a00645e7b38d5d29f7425367: Received BeginTabletCopySession request for tablet cc8e888d46d34a22b6d4409ec2ab53a9 from peer da3c4e948ff747678f18456c134d7ed4 ({username='slave'} at 127.7.39.132:34227)
I20240501 14:15:47.889717 19657 tablet_copy_service.cc:161] P ec8cde17a00645e7b38d5d29f7425367: Beginning new tablet copy session on tablet cc8e888d46d34a22b6d4409ec2ab53a9 from peer da3c4e948ff747678f18456c134d7ed4 at {username='slave'} at 127.7.39.132:34227: session id = da3c4e948ff747678f18456c134d7ed4-cc8e888d46d34a22b6d4409ec2ab53a9
I20240501 14:15:47.890662 19657 tablet_copy_source_session.cc:205] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 14:15:47.891454 12016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc8e888d46d34a22b6d4409ec2ab53a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:15:47.893326 12016 tablet_copy_client.cc:783] T cc8e888d46d34a22b6d4409ec2ab53a9 P da3c4e948ff747678f18456c134d7ed4: tablet copy: Starting download of 5 data blocks...
I20240501 14:15:47.897997 12016 tablet_copy_client.cc:647] T cc8e888d46d34a22b6d4409ec2ab53a9 P da3c4e948ff747678f18456c134d7ed4: tablet copy: Starting download of 1 WAL segments...
I20240501 14:15:47.899806 12016 tablet_copy_client.cc:523] T cc8e888d46d34a22b6d4409ec2ab53a9 P da3c4e948ff747678f18456c134d7ed4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:15:47.900846 12016 tablet_bootstrap.cc:492] T cc8e888d46d34a22b6d4409ec2ab53a9 P da3c4e948ff747678f18456c134d7ed4: Bootstrap starting.
I20240501 14:15:47.909809 12016 tablet_bootstrap.cc:492] T cc8e888d46d34a22b6d4409ec2ab53a9 P da3c4e948ff747678f18456c134d7ed4: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:15:47.910029 12016 tablet_bootstrap.cc:492] T cc8e888d46d34a22b6d4409ec2ab53a9 P da3c4e948ff747678f18456c134d7ed4: Bootstrap complete.
I20240501 14:15:47.910177 12016 ts_tablet_manager.cc:1367] T cc8e888d46d34a22b6d4409ec2ab53a9 P da3c4e948ff747678f18456c134d7ed4: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240501 14:15:47.910531 12016 raft_consensus.cc:348] T cc8e888d46d34a22b6d4409ec2ab53a9 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:15:47.910619 12016 raft_consensus.cc:729] T cc8e888d46d34a22b6d4409ec2ab53a9 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3c4e948ff747678f18456c134d7ed4, State: Initialized, Role: FOLLOWER
I20240501 14:15:47.913694 12016 consensus_queue.cc:260] T cc8e888d46d34a22b6d4409ec2ab53a9 P da3c4e948ff747678f18456c134d7ed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:15:47.913990 12016 ts_tablet_manager.cc:1398] T cc8e888d46d34a22b6d4409ec2ab53a9 P da3c4e948ff747678f18456c134d7ed4: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 14:15:47.915326 19657 tablet_copy_service.cc:342] P ec8cde17a00645e7b38d5d29f7425367: Request end of tablet copy session da3c4e948ff747678f18456c134d7ed4-cc8e888d46d34a22b6d4409ec2ab53a9 received from {username='slave'} at 127.7.39.132:34227
I20240501 14:15:47.915407 19657 tablet_copy_service.cc:433] P ec8cde17a00645e7b38d5d29f7425367: ending tablet copy session da3c4e948ff747678f18456c134d7ed4-cc8e888d46d34a22b6d4409ec2ab53a9 on tablet cc8e888d46d34a22b6d4409ec2ab53a9 with peer da3c4e948ff747678f18456c134d7ed4
I20240501 14:15:47.947505 21577 maintenance_manager.cc:419] P 778c78105092428bbcde75ab2cdb6a05: Scheduling UndoDeltaBlockGCOp(661aaf83e3474966a257fcc039ef4784): 4581 bytes on disk
I20240501 14:15:47.948256 21467 maintenance_manager.cc:643] P 778c78105092428bbcde75ab2cdb6a05: UndoDeltaBlockGCOp(661aaf83e3474966a257fcc039ef4784) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240501 14:15:47.951630 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:47.951856 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:47.960050 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:47.960727 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:47.961306 12114 ts_tablet_manager.cc:902] T 2dcca6e0e6374520bd4522c80c110e97 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Initiating tablet copy from peer ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201)
I20240501 14:15:47.961777 12114 tablet_copy_client.cc:308] T 2dcca6e0e6374520bd4522c80c110e97 P fb1e6bc2d07b4ce3b622a66f14c93bdc: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.131:45201
I20240501 14:15:47.962255 19657 tablet_copy_service.cc:140] P ec8cde17a00645e7b38d5d29f7425367: Received BeginTabletCopySession request for tablet 2dcca6e0e6374520bd4522c80c110e97 from peer fb1e6bc2d07b4ce3b622a66f14c93bdc ({username='slave'} at 127.7.39.129:47463)
I20240501 14:15:47.962349 19657 tablet_copy_service.cc:161] P ec8cde17a00645e7b38d5d29f7425367: Beginning new tablet copy session on tablet 2dcca6e0e6374520bd4522c80c110e97 from peer fb1e6bc2d07b4ce3b622a66f14c93bdc at {username='slave'} at 127.7.39.129:47463: session id = fb1e6bc2d07b4ce3b622a66f14c93bdc-2dcca6e0e6374520bd4522c80c110e97
I20240501 14:15:47.963352 19657 tablet_copy_source_session.cc:205] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 14:15:47.963696 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:47.964709 12114 ts_tablet_manager.cc:1875] T 2dcca6e0e6374520bd4522c80c110e97 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:15:47.967022 12114 ts_tablet_manager.cc:1888] T 2dcca6e0e6374520bd4522c80c110e97 P fb1e6bc2d07b4ce3b622a66f14c93bdc: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 14:15:47.967814 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:47.991813 12114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dcca6e0e6374520bd4522c80c110e97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:15:47.996953 12114 tablet_copy_client.cc:783] T 2dcca6e0e6374520bd4522c80c110e97 P fb1e6bc2d07b4ce3b622a66f14c93bdc: tablet copy: Starting download of 5 data blocks...
I20240501 14:15:48.001194 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:48.005143 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:48.010515 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:48.012470 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:48.020028 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:48.025491 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:48.029899 19423 maintenance_manager.cc:419] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Scheduling UndoDeltaBlockGCOp(85593473500b4d5eaf9f8b0bf2296777): 4834 bytes on disk
I20240501 14:15:48.030421 19313 maintenance_manager.cc:643] P fb1e6bc2d07b4ce3b622a66f14c93bdc: UndoDeltaBlockGCOp(85593473500b4d5eaf9f8b0bf2296777) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4}
I20240501 14:15:48.032269 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:48.037341 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:48.046198 12114 tablet_copy_client.cc:647] T 2dcca6e0e6374520bd4522c80c110e97 P fb1e6bc2d07b4ce3b622a66f14c93bdc: tablet copy: Starting download of 1 WAL segments...
I20240501 14:15:48.047372 12114 tablet_copy_client.cc:523] T 2dcca6e0e6374520bd4522c80c110e97 P fb1e6bc2d07b4ce3b622a66f14c93bdc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:15:48.049732 12114 tablet_bootstrap.cc:492] T 2dcca6e0e6374520bd4522c80c110e97 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Bootstrap starting.
I20240501 14:15:48.069278 19769 raft_consensus.cc:1203] T ffb4b6a2c19b497f8117fbbc9ea50082 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:15:48.083773 12250 ts_tablet_manager.cc:902] T fe554c3145de46cf8070efcecffbaedb P 43ce4a4c954f4a5f80954b38933c189d: Initiating tablet copy from peer ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201)
I20240501 14:15:48.084267 12250 tablet_copy_client.cc:308] T fe554c3145de46cf8070efcecffbaedb P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.131:45201
I20240501 14:15:48.085656 12114 tablet_bootstrap.cc:492] T 2dcca6e0e6374520bd4522c80c110e97 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:15:48.085925 12114 tablet_bootstrap.cc:492] T 2dcca6e0e6374520bd4522c80c110e97 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Bootstrap complete.
I20240501 14:15:48.086077 12114 ts_tablet_manager.cc:1367] T 2dcca6e0e6374520bd4522c80c110e97 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent bootstrapping tablet: real 0.036s user 0.005s sys 0.004s
I20240501 14:15:48.086537 12114 raft_consensus.cc:348] T 2dcca6e0e6374520bd4522c80c110e97 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:15:48.086658 12114 raft_consensus.cc:729] T 2dcca6e0e6374520bd4522c80c110e97 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1e6bc2d07b4ce3b622a66f14c93bdc, State: Initialized, Role: FOLLOWER
I20240501 14:15:48.086817 12114 consensus_queue.cc:260] T 2dcca6e0e6374520bd4522c80c110e97 P fb1e6bc2d07b4ce3b622a66f14c93bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:15:48.087134 12114 ts_tablet_manager.cc:1398] T 2dcca6e0e6374520bd4522c80c110e97 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:15:48.088295 19657 tablet_copy_service.cc:342] P ec8cde17a00645e7b38d5d29f7425367: Request end of tablet copy session fb1e6bc2d07b4ce3b622a66f14c93bdc-2dcca6e0e6374520bd4522c80c110e97 received from {username='slave'} at 127.7.39.129:47463
I20240501 14:15:48.088366 19657 tablet_copy_service.cc:433] P ec8cde17a00645e7b38d5d29f7425367: ending tablet copy session fb1e6bc2d07b4ce3b622a66f14c93bdc-2dcca6e0e6374520bd4522c80c110e97 on tablet 2dcca6e0e6374520bd4522c80c110e97 with peer fb1e6bc2d07b4ce3b622a66f14c93bdc
I20240501 14:15:48.091928 19657 tablet_copy_service.cc:140] P ec8cde17a00645e7b38d5d29f7425367: Received BeginTabletCopySession request for tablet fe554c3145de46cf8070efcecffbaedb from peer 43ce4a4c954f4a5f80954b38933c189d ({username='slave'} at 127.7.39.135:35443)
I20240501 14:15:48.092023 19657 tablet_copy_service.cc:161] P ec8cde17a00645e7b38d5d29f7425367: Beginning new tablet copy session on tablet fe554c3145de46cf8070efcecffbaedb from peer 43ce4a4c954f4a5f80954b38933c189d at {username='slave'} at 127.7.39.135:35443: session id = 43ce4a4c954f4a5f80954b38933c189d-fe554c3145de46cf8070efcecffbaedb
I20240501 14:15:48.092761 19657 tablet_copy_source_session.cc:205] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 14:15:48.093557 12250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe554c3145de46cf8070efcecffbaedb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:15:48.093578 19816 maintenance_manager.cc:419] P da3c4e948ff747678f18456c134d7ed4: Scheduling UndoDeltaBlockGCOp(cc8e888d46d34a22b6d4409ec2ab53a9): 4921 bytes on disk
I20240501 14:15:48.093998 19706 maintenance_manager.cc:643] P da3c4e948ff747678f18456c134d7ed4: UndoDeltaBlockGCOp(cc8e888d46d34a22b6d4409ec2ab53a9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240501 14:15:48.094683 19816 maintenance_manager.cc:419] P da3c4e948ff747678f18456c134d7ed4: Scheduling UndoDeltaBlockGCOp(d63afd081986473aa2715d074b123a93): 4921 bytes on disk
I20240501 14:15:48.095041 19706 maintenance_manager.cc:643] P da3c4e948ff747678f18456c134d7ed4: UndoDeltaBlockGCOp(d63afd081986473aa2715d074b123a93) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4}
I20240501 14:15:48.095634 12250 tablet_copy_client.cc:783] T fe554c3145de46cf8070efcecffbaedb P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Starting download of 5 data blocks...
I20240501 14:15:48.100471 12250 tablet_copy_client.cc:647] T fe554c3145de46cf8070efcecffbaedb P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Starting download of 1 WAL segments...
I20240501 14:15:48.101950 12250 tablet_copy_client.cc:523] T fe554c3145de46cf8070efcecffbaedb P 43ce4a4c954f4a5f80954b38933c189d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:15:48.103354 12250 tablet_bootstrap.cc:492] T fe554c3145de46cf8070efcecffbaedb P 43ce4a4c954f4a5f80954b38933c189d: Bootstrap starting.
I20240501 14:15:48.112488 12250 tablet_bootstrap.cc:492] T fe554c3145de46cf8070efcecffbaedb P 43ce4a4c954f4a5f80954b38933c189d: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:15:48.112740 12250 tablet_bootstrap.cc:492] T fe554c3145de46cf8070efcecffbaedb P 43ce4a4c954f4a5f80954b38933c189d: Bootstrap complete.
I20240501 14:15:48.112947 12250 ts_tablet_manager.cc:1367] T fe554c3145de46cf8070efcecffbaedb P 43ce4a4c954f4a5f80954b38933c189d: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20240501 14:15:48.113373 12250 raft_consensus.cc:348] T fe554c3145de46cf8070efcecffbaedb P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:15:48.113456 12250 raft_consensus.cc:729] T fe554c3145de46cf8070efcecffbaedb P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43ce4a4c954f4a5f80954b38933c189d, State: Initialized, Role: FOLLOWER
I20240501 14:15:48.113552 12250 consensus_queue.cc:260] T fe554c3145de46cf8070efcecffbaedb P 43ce4a4c954f4a5f80954b38933c189d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:15:48.113812 12250 ts_tablet_manager.cc:1398] T fe554c3145de46cf8070efcecffbaedb P 43ce4a4c954f4a5f80954b38933c189d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:15:48.114261 19657 tablet_copy_service.cc:342] P ec8cde17a00645e7b38d5d29f7425367: Request end of tablet copy session 43ce4a4c954f4a5f80954b38933c189d-fe554c3145de46cf8070efcecffbaedb received from {username='slave'} at 127.7.39.135:35443
I20240501 14:15:48.114331 19657 tablet_copy_service.cc:433] P ec8cde17a00645e7b38d5d29f7425367: ending tablet copy session 43ce4a4c954f4a5f80954b38933c189d-fe554c3145de46cf8070efcecffbaedb on tablet fe554c3145de46cf8070efcecffbaedb with peer 43ce4a4c954f4a5f80954b38933c189d
I20240501 14:15:48.128609 21522 raft_consensus.cc:1203] T 661aaf83e3474966a257fcc039ef4784 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:15:48.145936 19769 raft_consensus.cc:1203] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240501 14:15:48.147008 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:48.147420 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:48.152855 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:48.155387 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:15:48.159549 19219 catalog_manager.cc:4573] TS e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet fe554c3145de46cf8070efcecffbaedb: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
I20240501 14:15:48.163094 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:48.168845 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:48.172323 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:48.184595 12288 ts_tablet_manager.cc:902] T 11f64b59f83e458f8cf51fd5bcd36fee P 99a0be6ef2144b7fb247ba9d3f051d66: Initiating tablet copy from peer da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357)
I20240501 14:15:48.185110 12288 tablet_copy_client.cc:308] T 11f64b59f83e458f8cf51fd5bcd36fee P 99a0be6ef2144b7fb247ba9d3f051d66: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.132:35357
I20240501 14:15:48.185585 19789 tablet_copy_service.cc:140] P da3c4e948ff747678f18456c134d7ed4: Received BeginTabletCopySession request for tablet 11f64b59f83e458f8cf51fd5bcd36fee from peer 99a0be6ef2144b7fb247ba9d3f051d66 ({username='slave'} at 127.7.39.130:52281)
I20240501 14:15:48.185676 19789 tablet_copy_service.cc:161] P da3c4e948ff747678f18456c134d7ed4: Beginning new tablet copy session on tablet 11f64b59f83e458f8cf51fd5bcd36fee from peer 99a0be6ef2144b7fb247ba9d3f051d66 at {username='slave'} at 127.7.39.130:52281: session id = 99a0be6ef2144b7fb247ba9d3f051d66-11f64b59f83e458f8cf51fd5bcd36fee
I20240501 14:15:48.186883 19789 tablet_copy_source_session.cc:205] T 11f64b59f83e458f8cf51fd5bcd36fee P da3c4e948ff747678f18456c134d7ed4: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 14:15:48.191493 12288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11f64b59f83e458f8cf51fd5bcd36fee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:15:48.193607 12288 tablet_copy_client.cc:783] T 11f64b59f83e458f8cf51fd5bcd36fee P 99a0be6ef2144b7fb247ba9d3f051d66: tablet copy: Starting download of 5 data blocks...
I20240501 14:15:48.200590 12288 tablet_copy_client.cc:647] T 11f64b59f83e458f8cf51fd5bcd36fee P 99a0be6ef2144b7fb247ba9d3f051d66: tablet copy: Starting download of 1 WAL segments...
I20240501 14:15:48.201987 12288 tablet_copy_client.cc:523] T 11f64b59f83e458f8cf51fd5bcd36fee P 99a0be6ef2144b7fb247ba9d3f051d66: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:15:48.203207 12288 tablet_bootstrap.cc:492] T 11f64b59f83e458f8cf51fd5bcd36fee P 99a0be6ef2144b7fb247ba9d3f051d66: Bootstrap starting.
I20240501 14:15:48.210455 12288 tablet_bootstrap.cc:492] T 11f64b59f83e458f8cf51fd5bcd36fee P 99a0be6ef2144b7fb247ba9d3f051d66: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:15:48.210762 12288 tablet_bootstrap.cc:492] T 11f64b59f83e458f8cf51fd5bcd36fee P 99a0be6ef2144b7fb247ba9d3f051d66: Bootstrap complete.
I20240501 14:15:48.210937 12288 ts_tablet_manager.cc:1367] T 11f64b59f83e458f8cf51fd5bcd36fee P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.001s
I20240501 14:15:48.211334 12288 raft_consensus.cc:348] T 11f64b59f83e458f8cf51fd5bcd36fee P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:15:48.211434 12288 raft_consensus.cc:729] T 11f64b59f83e458f8cf51fd5bcd36fee P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a0be6ef2144b7fb247ba9d3f051d66, State: Initialized, Role: FOLLOWER
I20240501 14:15:48.211584 12288 consensus_queue.cc:260] T 11f64b59f83e458f8cf51fd5bcd36fee P 99a0be6ef2144b7fb247ba9d3f051d66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:15:48.211887 12288 ts_tablet_manager.cc:1398] T 11f64b59f83e458f8cf51fd5bcd36fee P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent starting tablet: real 0.001s user 0.000s sys 0.004s
I20240501 14:15:48.212575 19789 tablet_copy_service.cc:342] P da3c4e948ff747678f18456c134d7ed4: Request end of tablet copy session 99a0be6ef2144b7fb247ba9d3f051d66-11f64b59f83e458f8cf51fd5bcd36fee received from {username='slave'} at 127.7.39.130:52281
I20240501 14:15:48.212657 19789 tablet_copy_service.cc:433] P da3c4e948ff747678f18456c134d7ed4: ending tablet copy session 99a0be6ef2144b7fb247ba9d3f051d66-11f64b59f83e458f8cf51fd5bcd36fee on tablet 11f64b59f83e458f8cf51fd5bcd36fee with peer 99a0be6ef2144b7fb247ba9d3f051d66
I20240501 14:15:48.249188 19374 raft_consensus.cc:1203] T 85593473500b4d5eaf9f8b0bf2296777 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240501 14:15:48.257236 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:48.257714 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:48.258123 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:48.259083 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:48.260119 21446 maintenance_manager.cc:419] P 43ce4a4c954f4a5f80954b38933c189d: Scheduling UndoDeltaBlockGCOp(fe554c3145de46cf8070efcecffbaedb): 4921 bytes on disk
I20240501 14:15:48.260759 21336 maintenance_manager.cc:643] P 43ce4a4c954f4a5f80954b38933c189d: UndoDeltaBlockGCOp(fe554c3145de46cf8070efcecffbaedb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20240501 14:15:48.262065 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:48.262825 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:48.267261 19374 raft_consensus.cc:1203] T f69020bb6c244848a6b0a50fd2b4f3eb P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.11->[2.12-2.12] Dedup: 2.12->[]
I20240501 14:15:48.275487 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:48.281792 19423 maintenance_manager.cc:419] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Scheduling UndoDeltaBlockGCOp(2dcca6e0e6374520bd4522c80c110e97): 4920 bytes on disk
I20240501 14:15:48.282122 19313 maintenance_manager.cc:643] P fb1e6bc2d07b4ce3b622a66f14c93bdc: UndoDeltaBlockGCOp(2dcca6e0e6374520bd4522c80c110e97) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":42,"lbm_reads_lt_1ms":4}
I20240501 14:15:48.298277 12288 ts_tablet_manager.cc:902] T ad7c255baee74e629e2976a942a4174f P 99a0be6ef2144b7fb247ba9d3f051d66: Initiating tablet copy from peer fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847)
I20240501 14:15:48.298699 12288 tablet_copy_client.cc:308] T ad7c255baee74e629e2976a942a4174f P 99a0be6ef2144b7fb247ba9d3f051d66: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.129:35847
I20240501 14:15:48.299126 19396 tablet_copy_service.cc:140] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Received BeginTabletCopySession request for tablet ad7c255baee74e629e2976a942a4174f from peer 99a0be6ef2144b7fb247ba9d3f051d66 ({username='slave'} at 127.7.39.130:37261)
I20240501 14:15:48.299208 19396 tablet_copy_service.cc:161] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Beginning new tablet copy session on tablet ad7c255baee74e629e2976a942a4174f from peer 99a0be6ef2144b7fb247ba9d3f051d66 at {username='slave'} at 127.7.39.130:37261: session id = 99a0be6ef2144b7fb247ba9d3f051d66-ad7c255baee74e629e2976a942a4174f
I20240501 14:15:48.300189 19396 tablet_copy_source_session.cc:205] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 14:15:48.301256 19769 raft_consensus.cc:1203] T d63afd081986473aa2715d074b123a93 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240501 14:15:48.301401 12288 ts_tablet_manager.cc:1875] T ad7c255baee74e629e2976a942a4174f P 99a0be6ef2144b7fb247ba9d3f051d66: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:15:48.303640 12288 ts_tablet_manager.cc:1888] T ad7c255baee74e629e2976a942a4174f P 99a0be6ef2144b7fb247ba9d3f051d66: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240501 14:15:48.303817 12288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad7c255baee74e629e2976a942a4174f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:15:48.304857 12288 tablet_copy_client.cc:783] T ad7c255baee74e629e2976a942a4174f P 99a0be6ef2144b7fb247ba9d3f051d66: tablet copy: Starting download of 5 data blocks...
I20240501 14:15:48.310940 12288 tablet_copy_client.cc:647] T ad7c255baee74e629e2976a942a4174f P 99a0be6ef2144b7fb247ba9d3f051d66: tablet copy: Starting download of 1 WAL segments...
I20240501 14:15:48.312669 12288 tablet_copy_client.cc:523] T ad7c255baee74e629e2976a942a4174f P 99a0be6ef2144b7fb247ba9d3f051d66: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:15:48.313813 12288 tablet_bootstrap.cc:492] T ad7c255baee74e629e2976a942a4174f P 99a0be6ef2144b7fb247ba9d3f051d66: Bootstrap starting.
I20240501 14:15:48.324285 12288 tablet_bootstrap.cc:492] T ad7c255baee74e629e2976a942a4174f P 99a0be6ef2144b7fb247ba9d3f051d66: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:15:48.324540 12288 tablet_bootstrap.cc:492] T ad7c255baee74e629e2976a942a4174f P 99a0be6ef2144b7fb247ba9d3f051d66: Bootstrap complete.
I20240501 14:15:48.324726 12288 ts_tablet_manager.cc:1367] T ad7c255baee74e629e2976a942a4174f P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent bootstrapping tablet: real 0.011s user 0.011s sys 0.000s
I20240501 14:15:48.325423 12288 raft_consensus.cc:348] T ad7c255baee74e629e2976a942a4174f P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:15:48.325549 12288 raft_consensus.cc:729] T ad7c255baee74e629e2976a942a4174f P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a0be6ef2144b7fb247ba9d3f051d66, State: Initialized, Role: FOLLOWER
I20240501 14:15:48.325755 12288 consensus_queue.cc:260] T ad7c255baee74e629e2976a942a4174f P 99a0be6ef2144b7fb247ba9d3f051d66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:15:48.326186 12288 ts_tablet_manager.cc:1398] T ad7c255baee74e629e2976a942a4174f P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:15:48.326612 19396 tablet_copy_service.cc:342] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Request end of tablet copy session 99a0be6ef2144b7fb247ba9d3f051d66-ad7c255baee74e629e2976a942a4174f received from {username='slave'} at 127.7.39.130:37261
I20240501 14:15:48.326717 19396 tablet_copy_service.cc:433] P fb1e6bc2d07b4ce3b622a66f14c93bdc: ending tablet copy session 99a0be6ef2144b7fb247ba9d3f051d66-ad7c255baee74e629e2976a942a4174f on tablet ad7c255baee74e629e2976a942a4174f with peer 99a0be6ef2144b7fb247ba9d3f051d66
I20240501 14:15:48.383272 12114 ts_tablet_manager.cc:902] T 272eee3450cf4f48a4e1de2dcb28c0f7 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Initiating tablet copy from peer 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829)
I20240501 14:15:48.383796 12114 tablet_copy_client.cc:308] T 272eee3450cf4f48a4e1de2dcb28c0f7 P fb1e6bc2d07b4ce3b622a66f14c93bdc: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.130:41829
I20240501 14:15:48.384337 19527 tablet_copy_service.cc:140] P 99a0be6ef2144b7fb247ba9d3f051d66: Received BeginTabletCopySession request for tablet 272eee3450cf4f48a4e1de2dcb28c0f7 from peer fb1e6bc2d07b4ce3b622a66f14c93bdc ({username='slave'} at 127.7.39.129:53453)
I20240501 14:15:48.384446 19527 tablet_copy_service.cc:161] P 99a0be6ef2144b7fb247ba9d3f051d66: Beginning new tablet copy session on tablet 272eee3450cf4f48a4e1de2dcb28c0f7 from peer fb1e6bc2d07b4ce3b622a66f14c93bdc at {username='slave'} at 127.7.39.129:53453: session id = fb1e6bc2d07b4ce3b622a66f14c93bdc-272eee3450cf4f48a4e1de2dcb28c0f7
I20240501 14:15:48.385057 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:48.385331 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:48.385416 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:48.385802 19527 tablet_copy_source_session.cc:205] T 272eee3450cf4f48a4e1de2dcb28c0f7 P 99a0be6ef2144b7fb247ba9d3f051d66: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 14:15:48.386734 12114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 272eee3450cf4f48a4e1de2dcb28c0f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:15:48.389353 12114 tablet_copy_client.cc:783] T 272eee3450cf4f48a4e1de2dcb28c0f7 P fb1e6bc2d07b4ce3b622a66f14c93bdc: tablet copy: Starting download of 5 data blocks...
I20240501 14:15:48.393754 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:48.395602 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:48.397982 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:48.404789 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:48.427135 19769 raft_consensus.cc:1203] T cc8e888d46d34a22b6d4409ec2ab53a9 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240501 14:15:48.429484 12114 tablet_copy_client.cc:647] T 272eee3450cf4f48a4e1de2dcb28c0f7 P fb1e6bc2d07b4ce3b622a66f14c93bdc: tablet copy: Starting download of 1 WAL segments...
I20240501 14:15:48.430704 12114 tablet_copy_client.cc:523] T 272eee3450cf4f48a4e1de2dcb28c0f7 P fb1e6bc2d07b4ce3b622a66f14c93bdc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:15:48.432443 12114 tablet_bootstrap.cc:492] T 272eee3450cf4f48a4e1de2dcb28c0f7 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Bootstrap starting.
I20240501 14:15:48.440222 12114 tablet_bootstrap.cc:492] T 272eee3450cf4f48a4e1de2dcb28c0f7 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:15:48.440434 12114 tablet_bootstrap.cc:492] T 272eee3450cf4f48a4e1de2dcb28c0f7 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Bootstrap complete.
I20240501 14:15:48.440558 12114 ts_tablet_manager.cc:1367] T 272eee3450cf4f48a4e1de2dcb28c0f7 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240501 14:15:48.440877 12114 raft_consensus.cc:348] T 272eee3450cf4f48a4e1de2dcb28c0f7 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:15:48.440973 12114 raft_consensus.cc:729] T 272eee3450cf4f48a4e1de2dcb28c0f7 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1e6bc2d07b4ce3b622a66f14c93bdc, State: Initialized, Role: FOLLOWER
I20240501 14:15:48.441090 12114 consensus_queue.cc:260] T 272eee3450cf4f48a4e1de2dcb28c0f7 P fb1e6bc2d07b4ce3b622a66f14c93bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:15:48.441388 12114 ts_tablet_manager.cc:1398] T 272eee3450cf4f48a4e1de2dcb28c0f7 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:15:48.441936 19527 tablet_copy_service.cc:342] P 99a0be6ef2144b7fb247ba9d3f051d66: Request end of tablet copy session fb1e6bc2d07b4ce3b622a66f14c93bdc-272eee3450cf4f48a4e1de2dcb28c0f7 received from {username='slave'} at 127.7.39.129:53453
I20240501 14:15:48.442019 19527 tablet_copy_service.cc:433] P 99a0be6ef2144b7fb247ba9d3f051d66: ending tablet copy session fb1e6bc2d07b4ce3b622a66f14c93bdc-272eee3450cf4f48a4e1de2dcb28c0f7 on tablet 272eee3450cf4f48a4e1de2dcb28c0f7 with peer fb1e6bc2d07b4ce3b622a66f14c93bdc
I20240501 14:15:48.447361 19374 raft_consensus.cc:1203] T 2dcca6e0e6374520bd4522c80c110e97 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240501 14:15:48.450132 19374 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "410a7dd0dd734684a82875912966657d"
dest_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42856
I20240501 14:15:48.450248 19374 raft_consensus.cc:595] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Received request to transfer leadership
W20240501 14:15:48.452569 21071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 14:15:48.453686 21071 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): Couldn't send request to peer e6246b183293412494919f1e7efc9448. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 14:15:48.454828 19554 maintenance_manager.cc:419] P 99a0be6ef2144b7fb247ba9d3f051d66: Scheduling UndoDeltaBlockGCOp(11f64b59f83e458f8cf51fd5bcd36fee): 4834 bytes on disk
I20240501 14:15:48.455248 19444 maintenance_manager.cc:643] P 99a0be6ef2144b7fb247ba9d3f051d66: UndoDeltaBlockGCOp(11f64b59f83e458f8cf51fd5bcd36fee) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
I20240501 14:15:48.456298 19554 maintenance_manager.cc:419] P 99a0be6ef2144b7fb247ba9d3f051d66: Scheduling UndoDeltaBlockGCOp(ad7c255baee74e629e2976a942a4174f): 4834 bytes on disk
I20240501 14:15:48.456655 19444 maintenance_manager.cc:643] P 99a0be6ef2144b7fb247ba9d3f051d66: UndoDeltaBlockGCOp(ad7c255baee74e629e2976a942a4174f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
I20240501 14:15:48.496196 21398 raft_consensus.cc:1203] T fe554c3145de46cf8070efcecffbaedb P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240501 14:15:48.513309 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:48.514058 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:48.515535 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:48.517082 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:48.525538 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:48.529420 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:48.533728 19423 maintenance_manager.cc:419] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Scheduling UndoDeltaBlockGCOp(272eee3450cf4f48a4e1de2dcb28c0f7): 4834 bytes on disk
I20240501 14:15:48.534124 19313 maintenance_manager.cc:643] P fb1e6bc2d07b4ce3b622a66f14c93bdc: UndoDeltaBlockGCOp(272eee3450cf4f48a4e1de2dcb28c0f7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4}
I20240501 14:15:48.542860 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:48.611022 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:48.611764 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:48.615583 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:48.623072 21494 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:48.637384 19355 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:48.643358 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:48.648016 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:48.661252 11842 raft_consensus.cc:979] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc: : Instructing follower 778c78105092428bbcde75ab2cdb6a05 to start an election
I20240501 14:15:48.661336 11842 raft_consensus.cc:1067] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Signalling peer 778c78105092428bbcde75ab2cdb6a05 to start an election
I20240501 14:15:48.661756 21522 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "410a7dd0dd734684a82875912966657d"
dest_uuid: "778c78105092428bbcde75ab2cdb6a05"
from {username='slave'} at 127.7.39.129:49147
I20240501 14:15:48.661890 21522 raft_consensus.cc:482] T 410a7dd0dd734684a82875912966657d P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:15:48.661938 21522 raft_consensus.cc:3037] T 410a7dd0dd734684a82875912966657d P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:15:48.662955 21522 raft_consensus.cc:504] T 410a7dd0dd734684a82875912966657d P 778c78105092428bbcde75ab2cdb6a05 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:15:48.663357 21522 leader_election.cc:288] T 410a7dd0dd734684a82875912966657d P 778c78105092428bbcde75ab2cdb6a05 [CANDIDATE]: Term 2 election: Requested vote from peers fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847), da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357)
I20240501 14:15:48.663602 19374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "410a7dd0dd734684a82875912966657d" candidate_uuid: "778c78105092428bbcde75ab2cdb6a05" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc"
I20240501 14:15:48.663705 19374 raft_consensus.cc:3032] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:15:48.663733 19374 raft_consensus.cc:729] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fb1e6bc2d07b4ce3b622a66f14c93bdc, State: Running, Role: LEADER
I20240501 14:15:48.663864 19374 consensus_queue.cc:260] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:15:48.664016 19374 raft_consensus.cc:3037] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:15:48.664721 19374 raft_consensus.cc:2454] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 778c78105092428bbcde75ab2cdb6a05 in term 2.
I20240501 14:15:48.665019 21463 leader_election.cc:302] T 410a7dd0dd734684a82875912966657d P 778c78105092428bbcde75ab2cdb6a05 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 778c78105092428bbcde75ab2cdb6a05, fb1e6bc2d07b4ce3b622a66f14c93bdc; no voters:
I20240501 14:15:48.665201 8298 raft_consensus.cc:2781] T 410a7dd0dd734684a82875912966657d P 778c78105092428bbcde75ab2cdb6a05 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:15:48.665362 8298 raft_consensus.cc:686] T 410a7dd0dd734684a82875912966657d P 778c78105092428bbcde75ab2cdb6a05 [term 2 LEADER]: Becoming Leader. State: Replica: 778c78105092428bbcde75ab2cdb6a05, State: Running, Role: LEADER
I20240501 14:15:48.665491 8298 consensus_queue.cc:237] T 410a7dd0dd734684a82875912966657d P 778c78105092428bbcde75ab2cdb6a05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:15:48.667759 19231 catalog_manager.cc:5484] T 410a7dd0dd734684a82875912966657d P 778c78105092428bbcde75ab2cdb6a05 reported cstate change: term changed from 1 to 2, leader changed from fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) to 778c78105092428bbcde75ab2cdb6a05 (127.7.39.136). New cstate: current_term: 2 leader_uuid: "778c78105092428bbcde75ab2cdb6a05" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } }
I20240501 14:15:48.669777 19769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "410a7dd0dd734684a82875912966657d" candidate_uuid: "778c78105092428bbcde75ab2cdb6a05" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "da3c4e948ff747678f18456c134d7ed4"
I20240501 14:15:48.669874 19769 raft_consensus.cc:3037] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:15:48.670657 19769 raft_consensus.cc:2454] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 778c78105092428bbcde75ab2cdb6a05 in term 2.
I20240501 14:15:48.684840 19636 consensus_queue.cc:237] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } }
I20240501 14:15:48.685745 21522 raft_consensus.cc:1261] T 661aaf83e3474966a257fcc039ef4784 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:15:48.686072 11927 consensus_queue.cc:1035] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:15:48.687565 12331 raft_consensus.cc:2932] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:15:48.687958 21522 raft_consensus.cc:2932] T 661aaf83e3474966a257fcc039ef4784 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:15:48.690026 19231 catalog_manager.cc:5484] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 reported cstate change: config changed from index 7 to 8, VOTER da3c4e948ff747678f18456c134d7ed4 (127.7.39.132) evicted. New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } }
I20240501 14:15:48.692679 19749 tablet_service.cc:1514] Processing DeleteTablet for tablet 661aaf83e3474966a257fcc039ef4784 with delete_type TABLET_DATA_TOMBSTONED (TS da3c4e948ff747678f18456c134d7ed4 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:48108
I20240501 14:15:48.692976 12501 tablet_replica.cc:330] T 661aaf83e3474966a257fcc039ef4784 P da3c4e948ff747678f18456c134d7ed4: stopping tablet replica
I20240501 14:15:48.693167 12501 raft_consensus.cc:2229] T 661aaf83e3474966a257fcc039ef4784 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:15:48.693380 12501 raft_consensus.cc:2258] T 661aaf83e3474966a257fcc039ef4784 P da3c4e948ff747678f18456c134d7ed4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:15:48.693406 19636 consensus_queue.cc:237] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } }
I20240501 14:15:48.694029 12501 ts_tablet_manager.cc:1875] T 661aaf83e3474966a257fcc039ef4784 P da3c4e948ff747678f18456c134d7ed4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 14:15:48.694875 19571 consensus_peers.cc:485] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135:38751): Couldn't send request to peer 43ce4a4c954f4a5f80954b38933c189d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 661aaf83e3474966a257fcc039ef4784. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:15:48.695016 21522 raft_consensus.cc:1261] T 661aaf83e3474966a257fcc039ef4784 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Refusing update from remote peer ec8cde17a00645e7b38d5d29f7425367: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:15:48.695359 11927 consensus_queue.cc:1035] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:15:48.696822 12331 raft_consensus.cc:2932] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:15:48.697168 12501 ts_tablet_manager.cc:1888] T 661aaf83e3474966a257fcc039ef4784 P da3c4e948ff747678f18456c134d7ed4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 14:15:48.697256 12501 log.cc:1196] T 661aaf83e3474966a257fcc039ef4784 P da3c4e948ff747678f18456c134d7ed4: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal/wals/661aaf83e3474966a257fcc039ef4784
I20240501 14:15:48.697276 21522 raft_consensus.cc:2932] T 661aaf83e3474966a257fcc039ef4784 P 778c78105092428bbcde75ab2cdb6a05 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:15:48.697813 19220 catalog_manager.cc:4830] TS da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357): tablet 661aaf83e3474966a257fcc039ef4784 (table rebalance_test_table_1 [id=a9b1c37e923e49a7b30905624977117c]) successfully deleted
I20240501 14:15:48.698309 19219 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 661aaf83e3474966a257fcc039ef4784 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:15:48.699353 19231 catalog_manager.cc:5484] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 reported cstate change: config changed from index 8 to 9, VOTER 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135) added. New cstate: current_term: 1 leader_uuid: "ec8cde17a00645e7b38d5d29f7425367" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } }
I20240501 14:15:48.733728 19506 raft_consensus.cc:1203] T ad7c255baee74e629e2976a942a4174f P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240501 14:15:48.752056 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:48.754127 21494 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:48.754792 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:48.764647 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:48.775739 19355 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:15:48.778446 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:48.791148 19506 raft_consensus.cc:1203] T 11f64b59f83e458f8cf51fd5bcd36fee P 99a0be6ef2144b7fb247ba9d3f051d66 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240501 14:15:48.796689 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:48.863215 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:48.864373 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:48.871173 21494 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:48.872615 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:48.879118 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:48.906292 19355 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:15:48.907009 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:48.935760 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:48.937633 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:48.946648 21494 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:48.947870 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:48.955598 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:48.963678 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:48.970693 11769 raft_consensus.cc:998] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c: Attempting to remove follower e6246b183293412494919f1e7efc9448 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e6246b183293412494919f1e7efc9448 for more than 300 seconds (300.506s)
I20240501 14:15:48.970994 11769 consensus_queue.cc:237] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 3.11, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } }
I20240501 14:15:48.971210 19355 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:15:48.972302 21398 raft_consensus.cc:1261] T 814ba2247a314a4797aa0f9f624aaeb6 P 43ce4a4c954f4a5f80954b38933c189d [term 3 FOLLOWER]: Refusing update from remote peer 5bf81f3d986e428190d3f7727f36d96c: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240501 14:15:48.972685 12635 consensus_queue.cc:1035] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c [LEADER]: Connected to new peer: Peer: permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:15:48.974364 12635 raft_consensus.cc:2932] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c [term 3 LEADER]: Committing config change with OpId 3.12: config changed from index 10 to 12, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:15:48.974838 21398 raft_consensus.cc:2932] T 814ba2247a314a4797aa0f9f624aaeb6 P 43ce4a4c954f4a5f80954b38933c189d [term 3 FOLLOWER]: Committing config change with OpId 3.12: config changed from index 10 to 12, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:15:48.979404 19233 catalog_manager.cc:5484] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c reported cstate change: config changed from index 10 to 12, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) evicted. New cstate: current_term: 3 leader_uuid: "5bf81f3d986e428190d3f7727f36d96c" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } }
I20240501 14:15:48.996644 21137 consensus_queue.cc:237] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:15:48.998382 21398 raft_consensus.cc:1261] T 814ba2247a314a4797aa0f9f624aaeb6 P 43ce4a4c954f4a5f80954b38933c189d [term 3 FOLLOWER]: Refusing update from remote peer 5bf81f3d986e428190d3f7727f36d96c: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240501 14:15:48.998816 12635 consensus_queue.cc:1035] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c [LEADER]: Connected to new peer: Peer: permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:15:49.001281 12635 raft_consensus.cc:2932] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c [term 3 LEADER]: Committing config change with OpId 3.13: config changed from index 12 to 13, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } }
W20240501 14:15:49.001504 21070 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847): Couldn't send request to peer fb1e6bc2d07b4ce3b622a66f14c93bdc. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:15:49.001850 21398 raft_consensus.cc:2932] T 814ba2247a314a4797aa0f9f624aaeb6 P 43ce4a4c954f4a5f80954b38933c189d [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 12 to 13, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } }
I20240501 14:15:49.002748 19221 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 814ba2247a314a4797aa0f9f624aaeb6 with cas_config_opid_index 12: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:15:49.004137 19225 catalog_manager.cc:5484] T 814ba2247a314a4797aa0f9f624aaeb6 P 43ce4a4c954f4a5f80954b38933c189d reported cstate change: config changed from index 12 to 13, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) added. New cstate: current_term: 3 leader_uuid: "5bf81f3d986e428190d3f7727f36d96c" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } }
I20240501 14:15:49.004917 12644 raft_consensus.cc:998] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4: Attempting to remove follower e6246b183293412494919f1e7efc9448 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e6246b183293412494919f1e7efc9448 for more than 300 seconds (300.286s)
I20240501 14:15:49.005293 12644 consensus_queue.cc:237] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 3.7, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:15:49.006582 19375 raft_consensus.cc:1203] T 272eee3450cf4f48a4e1de2dcb28c0f7 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240501 14:15:49.007001 19374 raft_consensus.cc:1261] T 12ffa74f7a5b472aac57253693273cad P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 3 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
I20240501 14:15:49.007454 12047 consensus_queue.cc:1035] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 14:15:49.008281 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111) [suppressed 17 similar messages]
I20240501 14:15:49.009037 19636 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "85593473500b4d5eaf9f8b0bf2296777"
dest_uuid: "ec8cde17a00645e7b38d5d29f7425367"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46618
I20240501 14:15:49.009166 19636 raft_consensus.cc:595] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:15:49.010025 12644 raft_consensus.cc:2932] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 [term 3 LEADER]: Committing config change with OpId 3.8: config changed from index 6 to 8, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } }
I20240501 14:15:49.010499 19374 raft_consensus.cc:2932] T 12ffa74f7a5b472aac57253693273cad P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 3 FOLLOWER]: Committing config change with OpId 3.8: config changed from index 6 to 8, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } }
I20240501 14:15:49.013365 19233 catalog_manager.cc:5484] T 12ffa74f7a5b472aac57253693273cad P fb1e6bc2d07b4ce3b622a66f14c93bdc reported cstate change: config changed from index 6 to 8, VOTER e6246b183293412494919f1e7efc9448 (127.7.39.134) evicted. New cstate: current_term: 3 leader_uuid: "da3c4e948ff747678f18456c134d7ed4" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } }
I20240501 14:15:49.016255 19769 consensus_queue.cc:237] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 3.8, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:15:49.017518 19374 raft_consensus.cc:1261] T 12ffa74f7a5b472aac57253693273cad P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 3 FOLLOWER]: Refusing update from remote peer da3c4e948ff747678f18456c134d7ed4: Log matching property violated. Preceding OpId in replica: term: 3 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
W20240501 14:15:49.017521 19703 consensus_peers.cc:485] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 -> Peer 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829): Couldn't send request to peer 99a0be6ef2144b7fb247ba9d3f051d66. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 12ffa74f7a5b472aac57253693273cad. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:15:49.017877 12644 consensus_queue.cc:1035] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:15:49.019284 12047 raft_consensus.cc:2932] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 [term 3 LEADER]: Committing config change with OpId 3.9: config changed from index 8 to 9, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } }
I20240501 14:15:49.020673 19220 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 12ffa74f7a5b472aac57253693273cad with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:15:49.021317 19374 raft_consensus.cc:2932] T 12ffa74f7a5b472aac57253693273cad P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 3 FOLLOWER]: Committing config change with OpId 3.9: config changed from index 8 to 9, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } }
I20240501 14:15:49.021901 19233 catalog_manager.cc:5484] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4 reported cstate change: config changed from index 8 to 9, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) added. New cstate: current_term: 3 leader_uuid: "da3c4e948ff747678f18456c134d7ed4" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } }
I20240501 14:15:49.074532 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:49.077713 21378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:15:49.086722 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:49.090381 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:49.094890 21494 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:49.106122 19355 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:15:49.113480 19617 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:15:49.130033 12331 raft_consensus.cc:979] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367: : Instructing follower 43ce4a4c954f4a5f80954b38933c189d to start an election
I20240501 14:15:49.130167 11927 raft_consensus.cc:1067] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Signalling peer 43ce4a4c954f4a5f80954b38933c189d to start an election
I20240501 14:15:49.130597 21398 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "85593473500b4d5eaf9f8b0bf2296777"
dest_uuid: "43ce4a4c954f4a5f80954b38933c189d"
from {username='slave'} at 127.7.39.131:54975
I20240501 14:15:49.130736 21398 raft_consensus.cc:482] T 85593473500b4d5eaf9f8b0bf2296777 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:15:49.130789 21398 raft_consensus.cc:3037] T 85593473500b4d5eaf9f8b0bf2296777 P 43ce4a4c954f4a5f80954b38933c189d [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:15:49.131783 21398 raft_consensus.cc:504] T 85593473500b4d5eaf9f8b0bf2296777 P 43ce4a4c954f4a5f80954b38933c189d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:15:49.132380 21398 leader_election.cc:288] T 85593473500b4d5eaf9f8b0bf2296777 P 43ce4a4c954f4a5f80954b38933c189d [CANDIDATE]: Term 2 election: Requested vote from peers ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201), fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847)
I20240501 14:15:49.132730 19636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85593473500b4d5eaf9f8b0bf2296777" candidate_uuid: "43ce4a4c954f4a5f80954b38933c189d" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "ec8cde17a00645e7b38d5d29f7425367"
I20240501 14:15:49.132866 19636 raft_consensus.cc:3032] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:15:49.132838 19374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85593473500b4d5eaf9f8b0bf2296777" candidate_uuid: "43ce4a4c954f4a5f80954b38933c189d" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc"
I20240501 14:15:49.132910 19636 raft_consensus.cc:729] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ec8cde17a00645e7b38d5d29f7425367, State: Running, Role: LEADER
I20240501 14:15:49.132964 19374 raft_consensus.cc:3037] T 85593473500b4d5eaf9f8b0bf2296777 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:15:49.133062 19636 consensus_queue.cc:260] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:15:49.133284 19636 raft_consensus.cc:3037] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:15:49.134238 19374 raft_consensus.cc:2454] T 85593473500b4d5eaf9f8b0bf2296777 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43ce4a4c954f4a5f80954b38933c189d in term 2.
I20240501 14:15:49.134807 19636 raft_consensus.cc:2454] T 85593473500b4d5eaf9f8b0bf2296777 P ec8cde17a00645e7b38d5d29f7425367 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43ce4a4c954f4a5f80954b38933c189d in term 2.
I20240501 14:15:49.135159 21333 leader_election.cc:302] T 85593473500b4d5eaf9f8b0bf2296777 P 43ce4a4c954f4a5f80954b38933c189d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43ce4a4c954f4a5f80954b38933c189d, ec8cde17a00645e7b38d5d29f7425367; no voters:
I20240501 14:15:49.135349 11111 raft_consensus.cc:2781] T 85593473500b4d5eaf9f8b0bf2296777 P 43ce4a4c954f4a5f80954b38933c189d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:15:49.135512 11111 raft_consensus.cc:686] T 85593473500b4d5eaf9f8b0bf2296777 P 43ce4a4c954f4a5f80954b38933c189d [term 2 LEADER]: Becoming Leader. State: Replica: 43ce4a4c954f4a5f80954b38933c189d, State: Running, Role: LEADER
I20240501 14:15:49.135663 11111 consensus_queue.cc:237] T 85593473500b4d5eaf9f8b0bf2296777 P 43ce4a4c954f4a5f80954b38933c189d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:15:49.138128 19233 catalog_manager.cc:5484] T 85593473500b4d5eaf9f8b0bf2296777 P 43ce4a4c954f4a5f80954b38933c189d reported cstate change: term changed from 1 to 2, leader changed from ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131) to 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135). New cstate: current_term: 2 leader_uuid: "43ce4a4c954f4a5f80954b38933c189d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } }
I20240501 14:15:49.153520 12688 ts_tablet_manager.cc:902] T 661aaf83e3474966a257fcc039ef4784 P 43ce4a4c954f4a5f80954b38933c189d: Initiating tablet copy from peer ec8cde17a00645e7b38d5d29f7425367 (127.7.39.131:45201)
F20240501 14:15:49.153723 12688 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-10aa2feeb0324aa4a6221d2f2f3fc941.queue_time_us, tablet-download-pool-10aa2feeb0324aa4a6221d2f2f3fc941.queue_time_us) (tablet-download-pool-10aa2feeb0324aa4a6221d2f2f3fc941.run_cpu_time_us, tablet-download-pool-10aa2feeb0324aa4a6221d2f2f3fc941.run_cpu_time_us) (tablet-download-pool-10aa2feeb0324aa4a6221d2f2f3fc941.run_wall_time_us, tablet-download-pool-10aa2feeb0324aa4a6221d2f2f3fc941.run_wall_time_us) (tablet-download-pool-2dcca6e0e6374520bd4522c80c110e97.queue_time_us, tablet-download-pool-2dcca6e0e6374520bd4522c80c110e97.queue_time_us) (tablet-download-pool-2dcca6e0e6374520bd4522c80c110e97.run_cpu_time_us, tablet-download-pool-2dcca6e0e6374520bd4522c80c110e97.run_cpu_time_us) (tablet-download-pool-2dcca6e0e6374520bd4522c80c110e97.run_wall_time_us, tablet-download-pool-2dcca6e0e6374520bd4522c80c110e97.run_wall_time_us) (tablet-download-pool-661aaf83e3474966a257fcc039ef4784.queue_time_us, tablet-download-pool-661aaf83e3474966a257fcc039ef4784.queue_time_us) (tablet-download-pool-7eea58d4a33f422a9ada1e82d408577f.queue_time_us, tablet-download-pool-7eea58d4a33f422a9ada1e82d408577f.queue_time_us) (tablet-download-pool-7eea58d4a33f422a9ada1e82d408577f.run_cpu_time_us, tablet-download-pool-7eea58d4a33f422a9ada1e82d408577f.run_cpu_time_us) (tablet-download-pool-7eea58d4a33f422a9ada1e82d408577f.run_wall_time_us, tablet-download-pool-7eea58d4a33f422a9ada1e82d408577f.run_wall_time_us) (tablet-download-pool-814ba2247a314a4797aa0f9f624aaeb6.queue_time_us, tablet-download-pool-814ba2247a314a4797aa0f9f624aaeb6.queue_time_us) (tablet-download-pool-814ba2247a314a4797aa0f9f624aaeb6.run_cpu_time_us, tablet-download-pool-814ba2247a314a4797aa0f9f624aaeb6.run_cpu_time_us) (tablet-download-pool-814ba2247a314a4797aa0f9f624aaeb6.run_wall_time_us, tablet-download-pool-814ba2247a314a4797aa0f9f624aaeb6.run_wall_time_us) (tablet-download-pool-85593473500b4d5eaf9f8b0bf2296777.queue_time_us, tablet-download-pool-85593473500b4d5eaf9f8b0bf2296777.queue_time_us) (tablet-download-pool-85593473500b4d5eaf9f8b0bf2296777.run_cpu_time_us, tablet-download-pool-85593473500b4d5eaf9f8b0bf2296777.run_cpu_time_us) (tablet-download-pool-85593473500b4d5eaf9f8b0bf2296777.run_wall_time_us, tablet-download-pool-85593473500b4d5eaf9f8b0bf2296777.run_wall_time_us) (tablet-download-pool-98af8b06b9324e3fa7cdb898b7b9fab9.queue_time_us, tablet-download-pool-98af8b06b9324e3fa7cdb898b7b9fab9.queue_time_us) (tablet-download-pool-98af8b06b9324e3fa7cdb898b7b9fab9.run_cpu_time_us, tablet-download-pool-98af8b06b9324e3fa7cdb898b7b9fab9.run_cpu_time_us) (tablet-download-pool-98af8b06b9324e3fa7cdb898b7b9fab9.run_wall_time_us, tablet-download-pool-98af8b06b9324e3fa7cdb898b7b9fab9.run_wall_time_us) (tablet-download-pool-ad7c255baee74e629e2976a942a4174f.queue_time_us, tablet-download-pool-ad7c255baee74e629e2976a942a4174f.queue_time_us) (tablet-download-pool-ad7c255baee74e629e2976a942a4174f.run_cpu_time_us, tablet-download-pool-ad7c255baee74e629e2976a942a4174f.run_cpu_time_us) (tablet-download-pool-ad7c255baee74e629e2976a942a4174f.run_wall_time_us, tablet-download-pool-ad7c255baee74e629e2976a942a4174f.run_wall_time_us) (tablet-download-pool-b6be9a115fe14c098c29cad819849f95.queue_time_us, tablet-download-pool-b6be9a115fe14c098c29cad819849f95.queue_time_us) (tablet-download-pool-b6be9a115fe14c098c29cad819849f95.run_cpu_time_us, tablet-download-pool-b6be9a115fe14c098c29cad819849f95.run_cpu_time_us) (tablet-download-pool-b6be9a115fe14c098c29cad819849f95.run_wall_time_us, tablet-download-pool-b6be9a115fe14c098c29cad819849f95.run_wall_time_us) (tablet-download-pool-cc8e888d46d34a22b6d4409ec2ab53a9.queue_time_us, tablet-download-pool-cc8e888d46d34a22b6d4409ec2ab53a9.queue_time_us) (tablet-download-pool-cc8e888d46d34a22b6d4409ec2ab53a9.run_cpu_time_us, tablet-download-pool-cc8e888d46d34a22b6d4409ec2ab53a9.run_cpu_time_us) (tablet-download-pool-cc8e888d46d34a22b6d4409ec2ab53a9.run_wall_time_us, tablet-download-pool-cc8e888d46d34a22b6d4409ec2ab53a9.run_wall_time_us) (tablet-download-pool-d63afd081986473aa2715d074b123a93.queue_time_us, tablet-download-pool-d63afd081986473aa2715d074b123a93.queue_time_us) (tablet-download-pool-d63afd081986473aa2715d074b123a93.run_cpu_time_us, tablet-download-pool-d63afd081986473aa2715d074b123a93.run_cpu_time_us) (tablet-download-pool-d63afd081986473aa2715d074b123a93.run_wall_time_us, tablet-download-pool-d63afd081986473aa2715d074b123a93.run_wall_time_us) (tablet-download-pool-db0f01ee72b24ec9b84410cf94ef9759.queue_time_us, tablet-download-pool-db0f01ee72b24ec9b84410cf94ef9759.queue_time_us) (tablet-download-pool-db0f01ee72b24ec9b84410cf94ef9759.run_cpu_time_us, tablet-download-pool-db0f01ee72b24ec9b84410cf94ef9759.run_cpu_time_us) (tablet-download-pool-db0f01ee72b24ec9b84410cf94ef9759.run_wall_time_us, tablet-download-pool-db0f01ee72b24ec9b84410cf94ef9759.run_wall_time_us) (tablet-download-pool-f69020bb6c244848a6b0a50fd2b4f3eb.queue_time_us, tablet-download-pool-f69020bb6c244848a6b0a50fd2b4f3eb.queue_time_us) (tablet-download-pool-f69020bb6c244848a6b0a50fd2b4f3eb.run_cpu_time_us, tablet-download-pool-f69020bb6c244848a6b0a50fd2b4f3eb.run_cpu_time_us) (tablet-download-pool-f69020bb6c244848a6b0a50fd2b4f3eb.run_wall_time_us, tablet-download-pool-f69020bb6c244848a6b0a50fd2b4f3eb.run_wall_time_us) (tablet-download-pool-f6ca088ce3aa4b97aa54c8ce6d700921.queue_time_us, tablet-download-pool-f6ca088ce3aa4b97aa54c8ce6d700921.queue_time_us) (tablet-download-pool-f6ca088ce3aa4b97aa54c8ce6d700921.run_cpu_time_us, tablet-download-pool-f6ca088ce3aa4b97aa54c8ce6d700921.run_cpu_time_us) (tablet-download-pool-f6ca088ce3aa4b97aa54c8ce6d700921.run_wall_time_us, tablet-download-pool-f6ca088ce3aa4b97aa54c8ce6d700921.run_wall_time_us) (tablet-download-pool-fe554c3145de46cf8070efcecffbaedb.queue_time_us, tablet-download-pool-fe554c3145de46cf8070efcecffbaedb.queue_time_us) (tablet-download-pool-fe554c3145de46cf8070efcecffbaedb.run_cpu_time_us, tablet-download-pool-fe554c3145de46cf8070efcecffbaedb.run_cpu_time_us) (tablet-download-pool-fe554c3145de46cf8070efcecffbaedb.run_wall_time_us, tablet-download-pool-fe554c3145de46cf8070efcecffbaedb.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f0127b4bdcd google::LogMessage::Fail() at ??:0
@ 0x7f0127b4fb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f0127b4b7cc google::LogMessage::Flush() at ??:0
@ 0x7f0127b4cf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f0128fb5044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f0128f99f98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f0128f97efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f012bb4a77a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240501 14:15:49.172250 19769 raft_consensus.cc:1261] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4 [term 2 FOLLOWER]: Refusing update from remote peer 778c78105092428bbcde75ab2cdb6a05: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 14:15:49.172686 8298 consensus_queue.cc:1035] T 410a7dd0dd734684a82875912966657d P 778c78105092428bbcde75ab2cdb6a05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
@ 0x7f012bb53a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240501 14:15:49.176729 19374 raft_consensus.cc:1261] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 2 FOLLOWER]: Refusing update from remote peer 778c78105092428bbcde75ab2cdb6a05: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 14:15:49.177335 12704 consensus_queue.cc:1035] T 410a7dd0dd734684a82875912966657d P 778c78105092428bbcde75ab2cdb6a05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
@ 0x7f012bbd476f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f012bbe84f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f012bbd3226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7f012bbe2c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
W20240501 14:15:49.189386 19219 catalog_manager.cc:4573] TS e6246b183293412494919f1e7efc9448 (127.7.39.134:41911): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 814ba2247a314a4797aa0f9f624aaeb6: Network error: Client connection negotiation failed: client connection to 127.7.39.134:41911: connect: Connection refused (error 111)
@ 0x557f7655d75c std::function<>::operator()() at ??:0
@ 0x7f0128f9ce06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f0128f9d6ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f0128f9f03b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x557f7655d75c std::function<>::operator()() at ??:0
@ 0x7f0128f8c807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f01297ca6db start_thread at ??:0
@ 0x7f0126d0271f clone at ??:0
I20240501 14:15:49.267163 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:49.269460 21494 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:49.275978 19355 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:15:49.280201 19617 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:15:49.283320 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:49.299034 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:15:49.409621 19702 connection.cc:535] client connection to 127.7.39.135:38751 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:15:49.409808 19571 connection.cc:535] client connection to 127.7.39.135:38751 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:15:49.410892 19702 consensus_peers.cc:485] T 7eea58d4a33f422a9ada1e82d408577f P da3c4e948ff747678f18456c134d7ed4 -> Peer 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135:38751): Couldn't send request to peer 43ce4a4c954f4a5f80954b38933c189d. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.135:38751: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:15:49.410931 19571 consensus_peers.cc:462] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135:38751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.135:38751: connect: Connection refused (error 111)
W20240501 14:15:49.411000 19571 consensus_peers.cc:485] T cc8e888d46d34a22b6d4409ec2ab53a9 P ec8cde17a00645e7b38d5d29f7425367 -> Peer 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135:38751): Couldn't send request to peer 43ce4a4c954f4a5f80954b38933c189d. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.135:38751: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:15:49.425742 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:49.428211 19355 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:15:49.430126 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:49.431908 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:49.444046 21494 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:49.448951 19617 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 14:15:49.453855 21070 consensus_peers.cc:485] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c -> Peer 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135:38751): Couldn't send request to peer 43ce4a4c954f4a5f80954b38933c189d. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.135:38751: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:15:49.468232 19571 consensus_peers.cc:485] T fe554c3145de46cf8070efcecffbaedb P ec8cde17a00645e7b38d5d29f7425367 -> Peer 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135:38751): Couldn't send request to peer 43ce4a4c954f4a5f80954b38933c189d. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.135:38751: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:15:49.471583 12812 ts_tablet_manager.cc:902] T 12ffa74f7a5b472aac57253693273cad P 99a0be6ef2144b7fb247ba9d3f051d66: Initiating tablet copy from peer da3c4e948ff747678f18456c134d7ed4 (127.7.39.132:35357)
W20240501 14:15:49.471575 21070 consensus_peers.cc:485] T 98af8b06b9324e3fa7cdb898b7b9fab9 P 5bf81f3d986e428190d3f7727f36d96c -> Peer 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135:38751): Couldn't send request to peer 43ce4a4c954f4a5f80954b38933c189d. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.135:38751: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:15:49.472189 12812 tablet_copy_client.cc:308] T 12ffa74f7a5b472aac57253693273cad P 99a0be6ef2144b7fb247ba9d3f051d66: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.132:35357
I20240501 14:15:49.472604 19789 tablet_copy_service.cc:140] P da3c4e948ff747678f18456c134d7ed4: Received BeginTabletCopySession request for tablet 12ffa74f7a5b472aac57253693273cad from peer 99a0be6ef2144b7fb247ba9d3f051d66 ({username='slave'} at 127.7.39.130:52281)
I20240501 14:15:49.472712 19789 tablet_copy_service.cc:161] P da3c4e948ff747678f18456c134d7ed4: Beginning new tablet copy session on tablet 12ffa74f7a5b472aac57253693273cad from peer 99a0be6ef2144b7fb247ba9d3f051d66 at {username='slave'} at 127.7.39.130:52281: session id = 99a0be6ef2144b7fb247ba9d3f051d66-12ffa74f7a5b472aac57253693273cad
W20240501 14:15:49.473233 19440 consensus_peers.cc:485] T b6be9a115fe14c098c29cad819849f95 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135:38751): Couldn't send request to peer 43ce4a4c954f4a5f80954b38933c189d. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.135:38751: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:15:49.473742 19789 tablet_copy_source_session.cc:205] T 12ffa74f7a5b472aac57253693273cad P da3c4e948ff747678f18456c134d7ed4: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 14:15:49.474596 12812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12ffa74f7a5b472aac57253693273cad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:15:49.477144 12812 tablet_copy_client.cc:783] T 12ffa74f7a5b472aac57253693273cad P 99a0be6ef2144b7fb247ba9d3f051d66: tablet copy: Starting download of 5 data blocks...
I20240501 14:15:49.481775 12812 tablet_copy_client.cc:647] T 12ffa74f7a5b472aac57253693273cad P 99a0be6ef2144b7fb247ba9d3f051d66: tablet copy: Starting download of 1 WAL segments...
I20240501 14:15:49.483036 12812 tablet_copy_client.cc:523] T 12ffa74f7a5b472aac57253693273cad P 99a0be6ef2144b7fb247ba9d3f051d66: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:15:49.484480 12812 tablet_bootstrap.cc:492] T 12ffa74f7a5b472aac57253693273cad P 99a0be6ef2144b7fb247ba9d3f051d66: Bootstrap starting.
I20240501 14:15:49.486747 21522 consensus_queue.cc:237] T 410a7dd0dd734684a82875912966657d P 778c78105092428bbcde75ab2cdb6a05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } }
I20240501 14:15:49.487830 19769 raft_consensus.cc:1261] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4 [term 2 FOLLOWER]: Refusing update from remote peer 778c78105092428bbcde75ab2cdb6a05: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 14:15:49.488351 12704 consensus_queue.cc:1035] T 410a7dd0dd734684a82875912966657d P 778c78105092428bbcde75ab2cdb6a05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:15:49.489856 8298 raft_consensus.cc:2932] T 410a7dd0dd734684a82875912966657d P 778c78105092428bbcde75ab2cdb6a05 [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } }
I20240501 14:15:49.490242 19769 raft_consensus.cc:2932] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } }
I20240501 14:15:49.493677 19232 catalog_manager.cc:5484] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4 reported cstate change: config changed from index 11 to 13, VOTER fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129) evicted. New cstate: current_term: 2 leader_uuid: "778c78105092428bbcde75ab2cdb6a05" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } }
I20240501 14:15:49.495293 12812 tablet_bootstrap.cc:492] T 12ffa74f7a5b472aac57253693273cad P 99a0be6ef2144b7fb247ba9d3f051d66: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:15:49.495535 12812 tablet_bootstrap.cc:492] T 12ffa74f7a5b472aac57253693273cad P 99a0be6ef2144b7fb247ba9d3f051d66: Bootstrap complete.
I20240501 14:15:49.495659 12812 ts_tablet_manager.cc:1367] T 12ffa74f7a5b472aac57253693273cad P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20240501 14:15:49.495985 12812 raft_consensus.cc:348] T 12ffa74f7a5b472aac57253693273cad P 99a0be6ef2144b7fb247ba9d3f051d66 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:15:49.496095 12812 raft_consensus.cc:729] T 12ffa74f7a5b472aac57253693273cad P 99a0be6ef2144b7fb247ba9d3f051d66 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a0be6ef2144b7fb247ba9d3f051d66, State: Initialized, Role: FOLLOWER
I20240501 14:15:49.496482 12812 consensus_queue.cc:260] T 12ffa74f7a5b472aac57253693273cad P 99a0be6ef2144b7fb247ba9d3f051d66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 3.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:15:49.496896 12812 ts_tablet_manager.cc:1398] T 12ffa74f7a5b472aac57253693273cad P 99a0be6ef2144b7fb247ba9d3f051d66: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:15:49.497287 19355 tablet_service.cc:1514] Processing DeleteTablet for tablet 410a7dd0dd734684a82875912966657d with delete_type TABLET_DATA_TOMBSTONED (TS fb1e6bc2d07b4ce3b622a66f14c93bdc not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:42700
I20240501 14:15:49.497581 12838 tablet_replica.cc:330] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc: stopping tablet replica
I20240501 14:15:49.497717 19789 tablet_copy_service.cc:342] P da3c4e948ff747678f18456c134d7ed4: Request end of tablet copy session 99a0be6ef2144b7fb247ba9d3f051d66-12ffa74f7a5b472aac57253693273cad received from {username='slave'} at 127.7.39.130:52281
I20240501 14:15:49.497778 19789 tablet_copy_service.cc:433] P da3c4e948ff747678f18456c134d7ed4: ending tablet copy session 99a0be6ef2144b7fb247ba9d3f051d66-12ffa74f7a5b472aac57253693273cad on tablet 12ffa74f7a5b472aac57253693273cad with peer 99a0be6ef2144b7fb247ba9d3f051d66
W20240501 14:15:49.497804 19571 consensus_peers.cc:485] T db0f01ee72b24ec9b84410cf94ef9759 P ec8cde17a00645e7b38d5d29f7425367 -> Peer 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135:38751): Couldn't send request to peer 43ce4a4c954f4a5f80954b38933c189d. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.135:38751: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:15:49.498143 12838 raft_consensus.cc:2229] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:15:49.498322 12838 raft_consensus.cc:2258] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 2 FOLLOWER]: Raft consensus is shut down!
W20240501 14:15:49.498751 19309 consensus_peers.cc:485] T ad7c255baee74e629e2976a942a4174f P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135:38751): Couldn't send request to peer 43ce4a4c954f4a5f80954b38933c189d. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.135:38751: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:15:49.500551 12838 ts_tablet_manager.cc:1875] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:15:49.504103 12838 ts_tablet_manager.cc:1888] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.12
I20240501 14:15:49.504220 12838 log.cc:1196] T 410a7dd0dd734684a82875912966657d P fb1e6bc2d07b4ce3b622a66f14c93bdc: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal/wals/410a7dd0dd734684a82875912966657d
I20240501 14:15:49.504820 19218 catalog_manager.cc:4830] TS fb1e6bc2d07b4ce3b622a66f14c93bdc (127.7.39.129:35847): tablet 410a7dd0dd734684a82875912966657d (table rebalance_test_table_2 [id=6f6b0a378a484e5889d733ff982d9eb7]) successfully deleted
I20240501 14:15:49.506129 21522 consensus_queue.cc:237] T 410a7dd0dd734684a82875912966657d P 778c78105092428bbcde75ab2cdb6a05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } }
I20240501 14:15:49.507555 19769 raft_consensus.cc:1261] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4 [term 2 FOLLOWER]: Refusing update from remote peer 778c78105092428bbcde75ab2cdb6a05: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240501 14:15:49.508208 8298 consensus_queue.cc:1035] T 410a7dd0dd734684a82875912966657d P 778c78105092428bbcde75ab2cdb6a05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 14:15:49.509922 8298 raft_consensus.cc:2932] T 410a7dd0dd734684a82875912966657d P 778c78105092428bbcde75ab2cdb6a05 [term 2 LEADER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } }
W20240501 14:15:49.511751 21464 consensus_peers.cc:485] T 410a7dd0dd734684a82875912966657d P 778c78105092428bbcde75ab2cdb6a05 -> Peer 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829): Couldn't send request to peer 99a0be6ef2144b7fb247ba9d3f051d66. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:15:49.511757 19218 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 410a7dd0dd734684a82875912966657d with cas_config_opid_index 13: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:15:49.512241 19769 raft_consensus.cc:2932] T 410a7dd0dd734684a82875912966657d P da3c4e948ff747678f18456c134d7ed4 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } }
I20240501 14:15:49.512482 19232 catalog_manager.cc:5484] T 410a7dd0dd734684a82875912966657d P 778c78105092428bbcde75ab2cdb6a05 reported cstate change: config changed from index 13 to 14, VOTER 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130) added. New cstate: current_term: 2 leader_uuid: "778c78105092428bbcde75ab2cdb6a05" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "778c78105092428bbcde75ab2cdb6a05" member_type: VOTER last_known_addr { host: "127.7.39.136" port: 34895 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "99a0be6ef2144b7fb247ba9d3f051d66" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 41829 } } }
W20240501 14:15:49.532965 19440 consensus_peers.cc:485] T f6ca088ce3aa4b97aa54c8ce6d700921 P 99a0be6ef2144b7fb247ba9d3f051d66 -> Peer 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135:38751): Couldn't send request to peer 43ce4a4c954f4a5f80954b38933c189d. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.135:38751: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:15:49.543691 12877 ts_tablet_manager.cc:902] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Initiating tablet copy from peer 5bf81f3d986e428190d3f7727f36d96c (127.7.39.133:46883)
I20240501 14:15:49.544226 12877 tablet_copy_client.cc:308] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.133:46883
I20240501 14:15:49.544731 21157 tablet_copy_service.cc:140] P 5bf81f3d986e428190d3f7727f36d96c: Received BeginTabletCopySession request for tablet 814ba2247a314a4797aa0f9f624aaeb6 from peer fb1e6bc2d07b4ce3b622a66f14c93bdc ({username='slave'} at 127.7.39.129:53077)
I20240501 14:15:49.544850 21157 tablet_copy_service.cc:161] P 5bf81f3d986e428190d3f7727f36d96c: Beginning new tablet copy session on tablet 814ba2247a314a4797aa0f9f624aaeb6 from peer fb1e6bc2d07b4ce3b622a66f14c93bdc at {username='slave'} at 127.7.39.129:53077: session id = fb1e6bc2d07b4ce3b622a66f14c93bdc-814ba2247a314a4797aa0f9f624aaeb6
I20240501 14:15:49.546559 21157 tablet_copy_source_session.cc:205] T 814ba2247a314a4797aa0f9f624aaeb6 P 5bf81f3d986e428190d3f7727f36d96c: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 14:15:49.547505 12877 ts_tablet_manager.cc:1875] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:15:49.549569 12877 ts_tablet_manager.cc:1888] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.8
I20240501 14:15:49.549721 12877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 814ba2247a314a4797aa0f9f624aaeb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:15:49.550765 12877 tablet_copy_client.cc:783] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc: tablet copy: Starting download of 5 data blocks...
W20240501 14:15:49.551064 19571 consensus_peers.cc:485] T 10aa2feeb0324aa4a6221d2f2f3fc941 P ec8cde17a00645e7b38d5d29f7425367 -> Peer 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135:38751): Couldn't send request to peer 43ce4a4c954f4a5f80954b38933c189d. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.135:38751: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:15:49.557426 12877 tablet_copy_client.cc:647] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc: tablet copy: Starting download of 1 WAL segments...
W20240501 14:15:49.558255 19571 consensus_peers.cc:485] T 2dcca6e0e6374520bd4522c80c110e97 P ec8cde17a00645e7b38d5d29f7425367 -> Peer 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135:38751): Couldn't send request to peer 43ce4a4c954f4a5f80954b38933c189d. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.135:38751: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:15:49.559197 12877 tablet_copy_client.cc:523] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:15:49.560338 12877 tablet_bootstrap.cc:492] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Bootstrap starting.
I20240501 14:15:49.569929 12877 tablet_bootstrap.cc:492] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:15:49.570206 12877 tablet_bootstrap.cc:492] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Bootstrap complete.
I20240501 14:15:49.570381 12877 ts_tablet_manager.cc:1367] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.005s
I20240501 14:15:49.570783 12877 raft_consensus.cc:348] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:15:49.570899 12877 raft_consensus.cc:729] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1e6bc2d07b4ce3b622a66f14c93bdc, State: Initialized, Role: FOLLOWER
I20240501 14:15:49.571074 12877 consensus_queue.cc:260] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 3.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "43ce4a4c954f4a5f80954b38933c189d" member_type: VOTER last_known_addr { host: "127.7.39.135" port: 38751 } } peers { permanent_uuid: "5bf81f3d986e428190d3f7727f36d96c" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 46883 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }
I20240501 14:15:49.571348 12877 ts_tablet_manager.cc:1398] T 814ba2247a314a4797aa0f9f624aaeb6 P fb1e6bc2d07b4ce3b622a66f14c93bdc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:15:49.571853 21157 tablet_copy_service.cc:342] P 5bf81f3d986e428190d3f7727f36d96c: Request end of tablet copy session fb1e6bc2d07b4ce3b622a66f14c93bdc-814ba2247a314a4797aa0f9f624aaeb6 received from {username='slave'} at 127.7.39.129:53077
I20240501 14:15:49.571965 21157 tablet_copy_service.cc:433] P 5bf81f3d986e428190d3f7727f36d96c: ending tablet copy session fb1e6bc2d07b4ce3b622a66f14c93bdc-814ba2247a314a4797aa0f9f624aaeb6 on tablet 814ba2247a314a4797aa0f9f624aaeb6 with peer fb1e6bc2d07b4ce3b622a66f14c93bdc
I20240501 14:15:49.590821 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:49.591475 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:49.592401 21494 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:49.595168 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:49.597826 19617 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:15:49.624301 19355 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:15:49.625119 19223 maintenance_manager.cc:419] P e7ca182af9f547e7b7200a15751811d3: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 7323 bytes on disk
I20240501 14:15:49.627378 19222 maintenance_manager.cc:643] P e7ca182af9f547e7b7200a15751811d3: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.002s user 0.002s sys 0.000s Metrics: {"dirs.queue_time_us":167,"thread_start_us":95,"threads_started":1}
W20240501 14:15:49.653039 19309 consensus_peers.cc:485] T d63afd081986473aa2715d074b123a93 P fb1e6bc2d07b4ce3b622a66f14c93bdc -> Peer 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135:38751): Couldn't send request to peer 43ce4a4c954f4a5f80954b38933c189d. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.135:38751: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:15:49.656577 21583 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240501 14:10:32.584054 21584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:10:32.584381 21584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:10:32.590687 21584 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:10:33.638185 21584 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 14:10:34.102228 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a9b1c37e923e49a7b30905624977117c: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:34.102389 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 35d6f8fac5394197b987aa44dab206c1: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:34.148895 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b52269ef7c494f74a5e4f0d5cb3b8a47: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:34.149247 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6f6b0a378a484e5889d733ff982d9eb7: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:34.149317 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4d7621338643426d90fe03eb71654cb8: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:34.149430 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ed1a7ed87fa9480ba18a052130ecdc05: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:34.149516 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afc36d383e3e4795ad11804e9bebcb51: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:34.149551 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1fbbd460d13647c1a85b517926267158: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server e6246b183293412494919f1e7efc9448
W20240501 14:10:34.149590 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 35d6f8fac5394197b987aa44dab206c1: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:34.149621 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 151dfad0ef40470ab9c5203c42400b60: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:34.150292 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3866c35dd0e64d87b6049e5e14cb0d31: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:34.150421 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6f6b0a378a484e5889d733ff982d9eb7: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:34.150468 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a9b1c37e923e49a7b30905624977117c: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:34.150519 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4d7621338643426d90fe03eb71654cb8: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:34.150555 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afc36d383e3e4795ad11804e9bebcb51: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server e6246b183293412494919f1e7efc9448
I20240501 14:10:34.198734 21584 rebalancer_tool.cc:1074] tablet b6be9a115fe14c098c29cad819849f95: 'ec8cde17a00645e7b38d5d29f7425367' -> '43ce4a4c954f4a5f80954b38933c189d' move scheduled
I20240501 14:10:34.253064 21584 rebalancer_tool.cc:1084] tablet db0f01ee72b24ec9b84410cf94ef9759: 'fb1e6bc2d07b4ce3b622a66f14c93bdc' -> '778c78105092428bbcde75ab2cdb6a05' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } }.
Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:34.253172 21584 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:10:34.556771 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4d7621338643426d90fe03eb71654cb8: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server e6246b183293412494919f1e7efc9448
W20240501 14:10:34.556875 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afc36d383e3e4795ad11804e9bebcb51: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:34.557016 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 151dfad0ef40470ab9c5203c42400b60: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:34.557091 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afc36d383e3e4795ad11804e9bebcb51: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server e6246b183293412494919f1e7efc9448
W20240501 14:10:34.557138 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6f6b0a378a484e5889d733ff982d9eb7: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:34.557181 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a9b1c37e923e49a7b30905624977117c: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server e6246b183293412494919f1e7efc9448
W20240501 14:10:34.557222 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 35d6f8fac5394197b987aa44dab206c1: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:34.557262 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b52269ef7c494f74a5e4f0d5cb3b8a47: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:34.557315 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ed1a7ed87fa9480ba18a052130ecdc05: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:34.557400 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3866c35dd0e64d87b6049e5e14cb0d31: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:34.558432 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ed1a7ed87fa9480ba18a052130ecdc05: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server e6246b183293412494919f1e7efc9448
W20240501 14:10:34.558499 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1fbbd460d13647c1a85b517926267158: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:34.558542 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 151dfad0ef40470ab9c5203c42400b60: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server e6246b183293412494919f1e7efc9448
W20240501 14:10:34.558589 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4d7621338643426d90fe03eb71654cb8: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:34.558636 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6f6b0a378a484e5889d733ff982d9eb7: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:34.558677 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b52269ef7c494f74a5e4f0d5cb3b8a47: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:34.558720 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afc36d383e3e4795ad11804e9bebcb51: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server e6246b183293412494919f1e7efc9448
W20240501 14:10:34.558763 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a9b1c37e923e49a7b30905624977117c: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:34.558812 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3866c35dd0e64d87b6049e5e14cb0d31: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server 778c78105092428bbcde75ab2cdb6a05
I20240501 14:10:34.697527 21584 rebalancer_tool.cc:1074] tablet 55830424a98348e382759c54afdeabcd: 'fb1e6bc2d07b4ce3b622a66f14c93bdc' -> '778c78105092428bbcde75ab2cdb6a05' move scheduled
I20240501 14:10:34.746522 21584 rebalancer_tool.cc:1084] tablet 814ba2247a314a4797aa0f9f624aaeb6: '99a0be6ef2144b7fb247ba9d3f051d66' -> 'e6246b183293412494919f1e7efc9448' move ignored: Illegal state: tablet 814ba2247a314a4797aa0f9f624aaeb6: replica e6246b183293412494919f1e7efc9448 is already present
I20240501 14:10:34.746608 21584 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:10:35.536095 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afc36d383e3e4795ad11804e9bebcb51: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:35.536203 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4d7621338643426d90fe03eb71654cb8: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:35.536273 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 35d6f8fac5394197b987aa44dab206c1: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:35.536331 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1fbbd460d13647c1a85b517926267158: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:35.536381 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3866c35dd0e64d87b6049e5e14cb0d31: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:35.536438 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b52269ef7c494f74a5e4f0d5cb3b8a47: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:35.536532 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ed1a7ed87fa9480ba18a052130ecdc05: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:35.536572 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3866c35dd0e64d87b6049e5e14cb0d31: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:35.536612 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a9b1c37e923e49a7b30905624977117c: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:35.536648 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 151dfad0ef40470ab9c5203c42400b60: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:35.536684 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afc36d383e3e4795ad11804e9bebcb51: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server e6246b183293412494919f1e7efc9448
W20240501 14:10:35.536725 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6f6b0a378a484e5889d733ff982d9eb7: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:35.536787 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1fbbd460d13647c1a85b517926267158: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:35.536820 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 35d6f8fac5394197b987aa44dab206c1: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:35.536868 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ed1a7ed87fa9480ba18a052130ecdc05: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server e6246b183293412494919f1e7efc9448
W20240501 14:10:35.536900 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a9b1c37e923e49a7b30905624977117c: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:35.536933 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3866c35dd0e64d87b6049e5e14cb0d31: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 778c78105092428bbcde75ab2cdb6a05
I20240501 14:10:35.640116 21584 rebalancer_tool.cc:1084] tablet fe554c3145de46cf8070efcecffbaedb: '99a0be6ef2144b7fb247ba9d3f051d66' -> 'e6246b183293412494919f1e7efc9448' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240501 14:10:35.640197 21584 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:10:36.094105 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ed1a7ed87fa9480ba18a052130ecdc05: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:36.094194 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 35d6f8fac5394197b987aa44dab206c1: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server e6246b183293412494919f1e7efc9448
W20240501 14:10:36.094264 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3866c35dd0e64d87b6049e5e14cb0d31: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:36.094802 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4d7621338643426d90fe03eb71654cb8: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:36.094925 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6f6b0a378a484e5889d733ff982d9eb7: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server e6246b183293412494919f1e7efc9448
W20240501 14:10:36.095584 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a9b1c37e923e49a7b30905624977117c: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:36.095665 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1fbbd460d13647c1a85b517926267158: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server e6246b183293412494919f1e7efc9448
W20240501 14:10:36.095767 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 151dfad0ef40470ab9c5203c42400b60: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:36.095813 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ed1a7ed87fa9480ba18a052130ecdc05: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:36.096015 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b52269ef7c494f74a5e4f0d5cb3b8a47: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:36.096122 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3866c35dd0e64d87b6049e5e14cb0d31: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:36.096174 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afc36d383e3e4795ad11804e9bebcb51: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server e6246b183293412494919f1e7efc9448
W20240501 14:10:36.096204 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4d7621338643426d90fe03eb71654cb8: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:36.096232 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6f6b0a378a484e5889d733ff982d9eb7: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:36.096261 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1fbbd460d13647c1a85b517926267158: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:36.096302 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a9b1c37e923e49a7b30905624977117c: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 43ce4a4c954f4a5f80954b38933c189d
I20240501 14:10:36.293078 21584 rebalancer_tool.cc:1074] tablet ad7c255baee74e629e2976a942a4174f: '99a0be6ef2144b7fb247ba9d3f051d66' -> '' move scheduled
I20240501 14:10:36.489925 21584 rebalancer_tool.cc:1074] tablet 12ffa74f7a5b472aac57253693273cad: 'ec8cde17a00645e7b38d5d29f7425367' -> 'e6246b183293412494919f1e7efc9448' move scheduled
I20240501 14:10:36.601723 21584 rebalancer_tool.cc:1084] tablet 78e14dea8b6e4c0abd1c88dcd5b4e608: 'da3c4e948ff747678f18456c134d7ed4' -> '5bf81f3d986e428190d3f7727f36d96c' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240501 14:10:36.601817 21584 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:10:37.239434 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a9b1c37e923e49a7b30905624977117c: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:37.239567 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b52269ef7c494f74a5e4f0d5cb3b8a47: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:37.239650 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 35d6f8fac5394197b987aa44dab206c1: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server e6246b183293412494919f1e7efc9448
W20240501 14:10:37.239712 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a9b1c37e923e49a7b30905624977117c: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:37.239756 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6f6b0a378a484e5889d733ff982d9eb7: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:37.239884 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4d7621338643426d90fe03eb71654cb8: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 778c78105092428bbcde75ab2cdb6a05
I20240501 14:10:37.249234 21584 rebalancer_tool.cc:1074] tablet a9edf1764b55420ea5d519e07ae1d5e0: 'da3c4e948ff747678f18456c134d7ed4' -> '' move scheduled
I20240501 14:10:37.256794 21584 rebalancer_tool.cc:1074] tablet f6ca088ce3aa4b97aa54c8ce6d700921: 'fb1e6bc2d07b4ce3b622a66f14c93bdc' -> '' move scheduled
I20240501 14:10:37.307780 21584 rebalancer_tool.cc:1084] tablet ffb4b6a2c19b497f8117fbbc9ea50082: '99a0be6ef2144b7fb247ba9d3f051d66' -> '43ce4a4c954f4a5f80954b38933c189d' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240501 14:10:37.307889 21584 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:10:37.826771 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4d7621338643426d90fe03eb71654cb8: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:37.827011 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a9b1c37e923e49a7b30905624977117c: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server e6246b183293412494919f1e7efc9448
W20240501 14:10:37.841854 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6f6b0a378a484e5889d733ff982d9eb7: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:37.842244 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4d7621338643426d90fe03eb71654cb8: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:37.842334 21584 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afc36d383e3e4795ad11804e9bebcb51: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 43ce4a4c954f4a5f80954b38933c189d
I20240501 14:10:37.871371 21584 rebalancer_tool.cc:1074] tablet 814ba2247a314a4797aa0f9f624aaeb6: '99a0be6ef2144b7fb247ba9d3f051d66' -> '' move scheduled
I20240501 14:10:37.895391 21584 rebalancer_tool.cc:1074] tablet db0f01ee72b24ec9b84410cf94ef9759: 'da3c4e948ff747678f18456c134d7ed4' -> '' move scheduled
I20240501 14:10:39.699801 21584 rebalancer_tool.cc:1136] tablet b6be9a115fe14c098c29cad819849f95: 'ec8cde17a00645e7b38d5d29f7425367' -> '43ce4a4c954f4a5f80954b38933c189d' move completed: OK
I20240501 14:10:41.930720 21584 rebalancer_tool.cc:1136] tablet f6ca088ce3aa4b97aa54c8ce6d700921: 'fb1e6bc2d07b4ce3b622a66f14c93bdc' -> '' move completed: OK
I20240501 14:10:41.939910 21584 rebalancer_tool.cc:1074] tablet dab5e5722e2e4b0eaea73288d8dce265: 'ec8cde17a00645e7b38d5d29f7425367' -> '' move scheduled
I20240501 14:10:41.952474 21584 rebalancer_tool.cc:1136] tablet dab5e5722e2e4b0eaea73288d8dce265: 'ec8cde17a00645e7b38d5d29f7425367' -> '' move completed: OK
I20240501 14:10:41.968091 21584 rebalancer_tool.cc:1136] tablet db0f01ee72b24ec9b84410cf94ef9759: 'da3c4e948ff747678f18456c134d7ed4' -> '' move completed: OK
I20240501 14:10:41.983947 21584 rebalancer_tool.cc:1136] tablet 55830424a98348e382759c54afdeabcd: 'fb1e6bc2d07b4ce3b622a66f14c93bdc' -> '778c78105092428bbcde75ab2cdb6a05' move completed: OK
I20240501 14:10:42.001417 21584 rebalancer_tool.cc:1136] tablet ad7c255baee74e629e2976a942a4174f: '99a0be6ef2144b7fb247ba9d3f051d66' -> '' move completed: OK
I20240501 14:10:42.635061 21584 rebalancer_tool.cc:1136] tablet a9edf1764b55420ea5d519e07ae1d5e0: 'da3c4e948ff747678f18456c134d7ed4' -> '' move completed: OK
I20240501 14:10:42.648895 21584 rebalancer_tool.cc:1136] tablet 814ba2247a314a4797aa0f9f624aaeb6: '99a0be6ef2144b7fb247ba9d3f051d66' -> '' move completed: OK
I20240501 14:10:42.662003 21584 rebalancer_tool.cc:1074] tablet f69020bb6c244848a6b0a50fd2b4f3eb: 'fb1e6bc2d07b4ce3b622a66f14c93bdc' -> '' move scheduled
I20240501 14:10:42.695188 21584 rebalancer_tool.cc:1074] tablet d63afd081986473aa2715d074b123a93: 'da3c4e948ff747678f18456c134d7ed4' -> '' move scheduled
I20240501 14:10:42.797889 21584 rebalancer_tool.cc:1074] tablet 2dcca6e0e6374520bd4522c80c110e97: '99a0be6ef2144b7fb247ba9d3f051d66' -> '' move scheduled
I20240501 14:10:42.819675 21584 rebalancer_tool.cc:1074] tablet 0988c468757d49738765bdcb31794bd1: 'ec8cde17a00645e7b38d5d29f7425367' -> '' move scheduled
I20240501 14:10:42.903841 21584 rebalancer_tool.cc:1074] tablet cc8e888d46d34a22b6d4409ec2ab53a9: 'fb1e6bc2d07b4ce3b622a66f14c93bdc' -> '' move scheduled
I20240501 14:10:43.484959 21584 rebalancer_tool.cc:1136] tablet 0988c468757d49738765bdcb31794bd1: 'ec8cde17a00645e7b38d5d29f7425367' -> '' move completed: OK
I20240501 14:10:43.498203 21584 rebalancer_tool.cc:1136] tablet 2dcca6e0e6374520bd4522c80c110e97: '99a0be6ef2144b7fb247ba9d3f051d66' -> '' move completed: OK
I20240501 14:10:44.808784 21584 rebalancer_tool.cc:1136] tablet 12ffa74f7a5b472aac57253693273cad: 'ec8cde17a00645e7b38d5d29f7425367' -> 'e6246b183293412494919f1e7efc9448' move completed: OK
I20240501 14:10:44.831231 21584 rebalancer_tool.cc:1074] table
I20240501 14:15:49.713788 19554 maintenance_manager.cc:419] P 99a0be6ef2144b7fb247ba9d3f051d66: Scheduling UndoDeltaBlockGCOp(12ffa74f7a5b472aac57253693273cad): 4921 bytes on disk
I20240501 14:15:49.714160 19444 maintenance_manager.cc:643] P 99a0be6ef2144b7fb247ba9d3f051d66: UndoDeltaBlockGCOp(12ffa74f7a5b472aac57253693273cad) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4}
W20240501 14:15:49.729560 19571 consensus_peers.cc:462] T 661aaf83e3474966a257fcc039ef4784 P ec8cde17a00645e7b38d5d29f7425367 -> Peer 43ce4a4c954f4a5f80954b38933c189d (127.7.39.135:38751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.7.39.135:38751: connect: Connection refused (error 111)
I20240501 14:15:49.765383 21116 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:49.767210 21494 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:15:49.774111 19749 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:49.775198 19617 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:15:49.776930 19355 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:15:49.779825 19485 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:15:49.790127 19423 maintenance_manager.cc:419] P fb1e6bc2d07b4ce3b622a66f14c93bdc: Scheduling UndoDeltaBlockGCOp(814ba2247a314a4797aa0f9f624aaeb6): 4581 bytes on disk
I20240501 14:15:49.790493 19313 maintenance_manager.cc:643] P fb1e6bc2d07b4ce3b622a66f14c93bdc: UndoDeltaBlockGCOp(814ba2247a314a4797aa0f9f624aaeb6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
I20240501 14:15:49.825532 21581 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240501 14:10:32.534554 21585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:10:32.535037 21585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:10:32.554646 21585 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:10:33.641238 21585 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 14:10:34.175010 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3866c35dd0e64d87b6049e5e14cb0d31: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:34.175204 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1fbbd460d13647c1a85b517926267158: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:34.175292 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4d7621338643426d90fe03eb71654cb8: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:34.175360 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a9b1c37e923e49a7b30905624977117c: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server e6246b183293412494919f1e7efc9448
W20240501 14:10:34.175560 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 35d6f8fac5394197b987aa44dab206c1: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server e6246b183293412494919f1e7efc9448
W20240501 14:10:34.175668 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6f6b0a378a484e5889d733ff982d9eb7: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:34.175748 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afc36d383e3e4795ad11804e9bebcb51: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:34.175797 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 151dfad0ef40470ab9c5203c42400b60: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:34.175845 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ed1a7ed87fa9480ba18a052130ecdc05: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:34.175900 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b52269ef7c494f74a5e4f0d5cb3b8a47: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:34.175945 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 35d6f8fac5394197b987aa44dab206c1: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:34.176018 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1fbbd460d13647c1a85b517926267158: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:34.176065 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6f6b0a378a484e5889d733ff982d9eb7: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server e6246b183293412494919f1e7efc9448
W20240501 14:10:34.176110 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4d7621338643426d90fe03eb71654cb8: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:34.176160 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a9b1c37e923e49a7b30905624977117c: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 43ce4a4c954f4a5f80954b38933c189d
I20240501 14:10:34.245514 21585 rebalancer_tool.cc:1074] tablet 7eea58d4a33f422a9ada1e82d408577f: 'ec8cde17a00645e7b38d5d29f7425367' -> '43ce4a4c954f4a5f80954b38933c189d' move scheduled
I20240501 14:10:34.355171 21585 rebalancer_tool.cc:1074] tablet f69020bb6c244848a6b0a50fd2b4f3eb: '99a0be6ef2144b7fb247ba9d3f051d66' -> '5bf81f3d986e428190d3f7727f36d96c' move scheduled
I20240501 14:10:34.374809 21585 rebalancer_tool.cc:1084] tablet db0f01ee72b24ec9b84410cf94ef9759: 'fb1e6bc2d07b4ce3b622a66f14c93bdc' -> 'e6246b183293412494919f1e7efc9448' move ignored: Illegal state: tablet db0f01ee72b24ec9b84410cf94ef9759: replica e6246b183293412494919f1e7efc9448 is already present
I20240501 14:10:34.374912 21585 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:10:34.656214 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4d7621338643426d90fe03eb71654cb8: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server e6246b183293412494919f1e7efc9448
W20240501 14:10:34.656292 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afc36d383e3e4795ad11804e9bebcb51: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server e6246b183293412494919f1e7efc9448
W20240501 14:10:34.656353 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 151dfad0ef40470ab9c5203c42400b60: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:34.656515 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4d7621338643426d90fe03eb71654cb8: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:34.656558 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afc36d383e3e4795ad11804e9bebcb51: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:34.656599 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 151dfad0ef40470ab9c5203c42400b60: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server e6246b183293412494919f1e7efc9448
W20240501 14:10:34.656670 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ed1a7ed87fa9480ba18a052130ecdc05: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:34.656713 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 35d6f8fac5394197b987aa44dab206c1: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:34.656765 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b52269ef7c494f74a5e4f0d5cb3b8a47: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:34.656809 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a9b1c37e923e49a7b30905624977117c: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:34.656838 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3866c35dd0e64d87b6049e5e14cb0d31: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server e6246b183293412494919f1e7efc9448
W20240501 14:10:34.656868 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6f6b0a378a484e5889d733ff982d9eb7: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:34.656912 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1fbbd460d13647c1a85b517926267158: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:34.657003 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afc36d383e3e4795ad11804e9bebcb51: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server e6246b183293412494919f1e7efc9448
W20240501 14:10:34.657045 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 151dfad0ef40470ab9c5203c42400b60: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:34.657084 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ed1a7ed87fa9480ba18a052130ecdc05: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:34.657133 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b52269ef7c494f74a5e4f0d5cb3b8a47: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:34.657193 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 35d6f8fac5394197b987aa44dab206c1: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server e6246b183293412494919f1e7efc9448
I20240501 14:10:34.735675 21585 rebalancer_tool.cc:1084] tablet 11f64b59f83e458f8cf51fd5bcd36fee: 'fb1e6bc2d07b4ce3b622a66f14c93bdc' -> '778c78105092428bbcde75ab2cdb6a05' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } }.
Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec8cde17a00645e7b38d5d29f7425367" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45201 } } peers { permanent_uuid: "da3c4e948ff747678f18456c134d7ed4" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 35357 } } peers { permanent_uuid: "fb1e6bc2d07b4ce3b622a66f14c93bdc" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 35847 } } peers { permanent_uuid: "e6246b183293412494919f1e7efc9448" member_type: VOTER last_known_addr { host: "127.7.39.134" port: 41911 } }
I20240501 14:10:34.735759 21585 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:10:35.178381 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afc36d383e3e4795ad11804e9bebcb51: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:35.178541 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 151dfad0ef40470ab9c5203c42400b60: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:35.178589 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b52269ef7c494f74a5e4f0d5cb3b8a47: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:35.178645 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3866c35dd0e64d87b6049e5e14cb0d31: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:35.178689 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ed1a7ed87fa9480ba18a052130ecdc05: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server e6246b183293412494919f1e7efc9448
W20240501 14:10:35.178748 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4d7621338643426d90fe03eb71654cb8: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:35.178838 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 35d6f8fac5394197b987aa44dab206c1: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server e6246b183293412494919f1e7efc9448
W20240501 14:10:35.178884 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a9b1c37e923e49a7b30905624977117c: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:35.178925 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3866c35dd0e64d87b6049e5e14cb0d31: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:35.178969 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1fbbd460d13647c1a85b517926267158: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:35.179019 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afc36d383e3e4795ad11804e9bebcb51: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:35.179076 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6f6b0a378a484e5889d733ff982d9eb7: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server e6246b183293412494919f1e7efc9448
W20240501 14:10:35.179111 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 151dfad0ef40470ab9c5203c42400b60: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:35.179152 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b52269ef7c494f74a5e4f0d5cb3b8a47: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:35.179211 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4d7621338643426d90fe03eb71654cb8: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server e6246b183293412494919f1e7efc9448
W20240501 14:10:35.179255 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a9b1c37e923e49a7b30905624977117c: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:35.179307 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ed1a7ed87fa9480ba18a052130ecdc05: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:35.179353 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 35d6f8fac5394197b987aa44dab206c1: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:35.179396 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3866c35dd0e64d87b6049e5e14cb0d31: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:35.179445 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afc36d383e3e4795ad11804e9bebcb51: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 43ce4a4c954f4a5f80954b38933c189d
I20240501 14:10:35.450935 21585 rebalancer_tool.cc:1074] tablet 2dcca6e0e6374520bd4522c80c110e97: 'fb1e6bc2d07b4ce3b622a66f14c93bdc' -> 'e6246b183293412494919f1e7efc9448' move scheduled
I20240501 14:10:35.495225 21585 rebalancer_tool.cc:1084] tablet 272eee3450cf4f48a4e1de2dcb28c0f7: 'ec8cde17a00645e7b38d5d29f7425367' -> '778c78105092428bbcde75ab2cdb6a05' move ignored: Illegal state: tablet 272eee3450cf4f48a4e1de2dcb28c0f7: replica 778c78105092428bbcde75ab2cdb6a05 is already present
I20240501 14:10:35.495312 21585 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:10:35.854228 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1fbbd460d13647c1a85b517926267158: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server e6246b183293412494919f1e7efc9448
W20240501 14:10:35.854542 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a9b1c37e923e49a7b30905624977117c: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:35.854630 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3866c35dd0e64d87b6049e5e14cb0d31: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:35.854696 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a9b1c37e923e49a7b30905624977117c: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server e6246b183293412494919f1e7efc9448
W20240501 14:10:35.854756 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3866c35dd0e64d87b6049e5e14cb0d31: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:35.854815 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b52269ef7c494f74a5e4f0d5cb3b8a47: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:35.854866 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1fbbd460d13647c1a85b517926267158: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server e6246b183293412494919f1e7efc9448
W20240501 14:10:35.854910 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ed1a7ed87fa9480ba18a052130ecdc05: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:35.854957 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afc36d383e3e4795ad11804e9bebcb51: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:35.855001 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4d7621338643426d90fe03eb71654cb8: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:35.855087 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 151dfad0ef40470ab9c5203c42400b60: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:35.855144 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6f6b0a378a484e5889d733ff982d9eb7: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:35.855192 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 35d6f8fac5394197b987aa44dab206c1: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:35.855258 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3866c35dd0e64d87b6049e5e14cb0d31: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:35.855307 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b52269ef7c494f74a5e4f0d5cb3b8a47: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:35.855370 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1fbbd460d13647c1a85b517926267158: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 5bf81f3d986e428190d3f7727f36d96c
I20240501 14:10:35.880571 21585 rebalancer_tool.cc:1074] tablet a9edf1764b55420ea5d519e07ae1d5e0: 'da3c4e948ff747678f18456c134d7ed4' -> '' move scheduled
I20240501 14:10:35.984977 21585 rebalancer_tool.cc:1084] tablet bb291a0c743c4e638f79da36cc77faa3: 'ec8cde17a00645e7b38d5d29f7425367' -> '778c78105092428bbcde75ab2cdb6a05' move ignored: Illegal state: tablet bb291a0c743c4e638f79da36cc77faa3: replica 778c78105092428bbcde75ab2cdb6a05 is already present
I20240501 14:10:35.985073 21585 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:10:36.458441 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3866c35dd0e64d87b6049e5e14cb0d31: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:36.458539 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6f6b0a378a484e5889d733ff982d9eb7: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:36.458590 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a9b1c37e923e49a7b30905624977117c: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server e6246b183293412494919f1e7efc9448
W20240501 14:10:36.458642 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a9b1c37e923e49a7b30905624977117c: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:36.458678 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b52269ef7c494f74a5e4f0d5cb3b8a47: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:36.458707 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afc36d383e3e4795ad11804e9bebcb51: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:36.458758 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 151dfad0ef40470ab9c5203c42400b60: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:36.458807 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1fbbd460d13647c1a85b517926267158: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:36.458855 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a9b1c37e923e49a7b30905624977117c: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:36.458885 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4d7621338643426d90fe03eb71654cb8: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:36.458914 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b52269ef7c494f74a5e4f0d5cb3b8a47: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 43ce4a4c954f4a5f80954b38933c189d
I20240501 14:10:36.529489 21585 rebalancer_tool.cc:1074] tablet dadab886b03540579d138cdb5b25e286: '99a0be6ef2144b7fb247ba9d3f051d66' -> '778c78105092428bbcde75ab2cdb6a05' move scheduled
I20240501 14:10:36.564754 21585 rebalancer_tool.cc:1074] tablet 55830424a98348e382759c54afdeabcd: 'fb1e6bc2d07b4ce3b622a66f14c93bdc' -> '' move scheduled
I20240501 14:10:36.579672 21585 rebalancer_tool.cc:1084] tablet dab5e5722e2e4b0eaea73288d8dce265: 'ec8cde17a00645e7b38d5d29f7425367' -> '5bf81f3d986e428190d3f7727f36d96c' move ignored: Illegal state: tablet dab5e5722e2e4b0eaea73288d8dce265: replica 5bf81f3d986e428190d3f7727f36d96c is already present
I20240501 14:10:36.579757 21585 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:10:37.196573 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a9b1c37e923e49a7b30905624977117c: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:37.196856 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3866c35dd0e64d87b6049e5e14cb0d31: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:37.196940 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b52269ef7c494f74a5e4f0d5cb3b8a47: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server 43ce4a4c954f4a5f80954b38933c189d
W20240501 14:10:37.197002 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6f6b0a378a484e5889d733ff982d9eb7: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:37.197049 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4d7621338643426d90fe03eb71654cb8: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to server e6246b183293412494919f1e7efc9448
W20240501 14:10:37.197161 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a9b1c37e923e49a7b30905624977117c: could not find any suitable replica to move from server da3c4e948ff747678f18456c134d7ed4 to server 5bf81f3d986e428190d3f7727f36d96c
W20240501 14:10:37.197223 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ed1a7ed87fa9480ba18a052130ecdc05: could not find any suitable replica to move from server 99a0be6ef2144b7fb247ba9d3f051d66 to server e6246b183293412494919f1e7efc9448
W20240501 14:10:37.197273 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 151dfad0ef40470ab9c5203c42400b60: could not find any suitable replica to move from server fb1e6bc2d07b4ce3b622a66f14c93bdc to server 778c78105092428bbcde75ab2cdb6a05
W20240501 14:10:37.197362 21585 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1fbbd460d13647c1a85b517926267158: could not find any suitable replica to move from server ec8cde17a00645e7b38d5d29f7425367 to serv
E20240501 14:15:49.825973 7326 external_mini_cluster.cc:943] Process with UUID e6246b183293412494919f1e7efc9448 has crashed
E20240501 14:15:49.826072 7326 external_mini_cluster.cc:943] Process with UUID 43ce4a4c954f4a5f80954b38933c189d has crashed
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/mini-cluster/external_mini_cluster.cc:947: Failure
Expected equality of these values:
0
num_crashes
Which is: 2
At least one process crashed
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tools/rebalancer_tool-test.cc:1284: Failure
Expected: cluster_->AssertNoCrashes() doesn't generate new fatal failures in the current thread.
Actual: it does.
I20240501 14:15:49.826777 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 19294
I20240501 14:15:49.835031 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 19425
W20240501 14:15:49.842797 19703 connection.cc:535] client connection to 127.7.39.130:41829 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:15:49.843173 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 19556
W20240501 14:15:49.843966 19703 consensus_peers.cc:485] T 7eea58d4a33f422a9ada1e82d408577f P da3c4e948ff747678f18456c134d7ed4 -> Peer 99a0be6ef2144b7fb247ba9d3f051d66 (127.7.39.130:41829): Couldn't send request to peer 99a0be6ef2144b7fb247ba9d3f051d66. Status: Network error: Client connection negotiation failed: client connection to 127.7.39.130:41829: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:15:49.852600 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 19687
I20240501 14:15:49.860266 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 21055
I20240501 14:15:49.867291 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 21448
I20240501 14:15:49.874336 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 19203
2024-05-01T14:15:49Z chronyd exiting
I20240501 14:15:49.893074 7326 test_util.cc:175] -----------------------------------------------
I20240501 14:15:49.893128 7326 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714572528121239-7326-0
[ FAILED ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/0, where GetParam() = 4-byte object <00-00 00-00> (329141 ms)
[----------] 1 test from ConcurrentRebalancersTest (329141 ms total)
[----------] 1 test from RebalancerAndSingleReplicaTablets
[ RUN ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/0
I20240501 14:15:49.893877 7326 test_util.cc:268] Using random seed: 530494194
I20240501 14:15:49.894627 7326 ts_itest-base.cc:112] Starting cluster with:
I20240501 14:15:49.894670 7326 ts_itest-base.cc:113] --------------
I20240501 14:15:49.894694 7326 ts_itest-base.cc:114] 4 tablet servers
I20240501 14:15:49.894712 7326 ts_itest-base.cc:115] 1 replicas per TS
I20240501 14:15:49.894742 7326 ts_itest-base.cc:116] --------------
2024-05-01T14:15:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T14:15:49Z Disabled control of system clock
I20240501 14:15:49.904124 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.39.190:37333
--webserver_interface=127.7.39.190
--webserver_port=0
--builtin_ntp_servers=127.7.39.148:33663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.39.190:37333
--raft_prepare_replacement_before_eviction=false
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false with env {}
W20240501 14:15:50.007949 12960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:15:50.008155 12960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:15:50.008201 12960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:15:50.011101 12960 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:15:50.011183 12960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 14:15:50.011207 12960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:15:50.011224 12960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 14:15:50.011246 12960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 14:15:50.014676 12960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:33663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.7.39.190:37333
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.39.190:37333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.7.39.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.12960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:15:50.015437 12960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:15:50.016425 12960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:15:50.023562 12966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:15:50.024019 12968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:15:50.024734 12965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:15:50.025352 12960 server_base.cc:1034] running on GCE node
I20240501 14:15:50.025791 12960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:15:50.026625 12960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:15:50.027767 12960 hybrid_clock.cc:648] HybridClock initialized: now 1714572950027757 us; error 23 us; skew 500 ppm
I20240501 14:15:50.029567 12960 webserver.cc:460] Webserver started at http://127.7.39.190:44067/ using document root <none> and password file <none>
I20240501 14:15:50.030097 12960 fs_manager.cc:359] Metadata directory not provided
I20240501 14:15:50.030153 12960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:15:50.030330 12960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:15:50.031931 12960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "4e0fceaf25e84af7832884004d8d5cb7"
format_stamp: "Formatted at 2024-05-01 14:15:50 on dist-test-slave-rnfw"
I20240501 14:15:50.032372 12960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "4e0fceaf25e84af7832884004d8d5cb7"
format_stamp: "Formatted at 2024-05-01 14:15:50 on dist-test-slave-rnfw"
I20240501 14:15:50.035530 12960 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:15:50.037432 12974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:15:50.038340 12960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:15:50.038441 12960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/wal
uuid: "4e0fceaf25e84af7832884004d8d5cb7"
format_stamp: "Formatted at 2024-05-01 14:15:50 on dist-test-slave-rnfw"
I20240501 14:15:50.038514 12960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:15:50.052455 12960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:15:50.052969 12960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:15:50.053109 12960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:15:50.059214 12960 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.190:37333
I20240501 14:15:50.059216 13026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.190:37333 every 8 connection(s)
I20240501 14:15:50.060245 12960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 14:15:50.063485 13027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:15:50.068899 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 12960
I20240501 14:15:50.069020 7326 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 14:15:50.069411 13027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e0fceaf25e84af7832884004d8d5cb7: Bootstrap starting.
I20240501 14:15:50.071678 13027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e0fceaf25e84af7832884004d8d5cb7: Neither blocks nor log segments found. Creating new log.
I20240501 14:15:50.072456 13027 log.cc:824] T 00000000000000000000000000000000 P 4e0fceaf25e84af7832884004d8d5cb7: Log is configured to *not* fsync() on all Append() calls
I20240501 14:15:50.074051 13027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e0fceaf25e84af7832884004d8d5cb7: No bootstrap required, opened a new log
I20240501 14:15:50.076861 13027 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e0fceaf25e84af7832884004d8d5cb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e0fceaf25e84af7832884004d8d5cb7" member_type: VOTER last_known_addr { host: "127.7.39.190" port: 37333 } }
I20240501 14:15:50.077056 13027 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e0fceaf25e84af7832884004d8d5cb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:15:50.077102 13027 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e0fceaf25e84af7832884004d8d5cb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e0fceaf25e84af7832884004d8d5cb7, State: Initialized, Role: FOLLOWER
I20240501 14:15:50.077488 13027 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e0fceaf25e84af7832884004d8d5cb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e0fceaf25e84af7832884004d8d5cb7" member_type: VOTER last_known_addr { host: "127.7.39.190" port: 37333 } }
I20240501 14:15:50.077594 13027 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e0fceaf25e84af7832884004d8d5cb7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:15:50.077684 13027 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e0fceaf25e84af7832884004d8d5cb7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:15:50.077800 13027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e0fceaf25e84af7832884004d8d5cb7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:15:50.078485 13027 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e0fceaf25e84af7832884004d8d5cb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e0fceaf25e84af7832884004d8d5cb7" member_type: VOTER last_known_addr { host: "127.7.39.190" port: 37333 } }
I20240501 14:15:50.078769 13027 leader_election.cc:302] T 00000000000000000000000000000000 P 4e0fceaf25e84af7832884004d8d5cb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e0fceaf25e84af7832884004d8d5cb7; no voters:
I20240501 14:15:50.079010 13027 leader_election.cc:288] T 00000000000000000000000000000000 P 4e0fceaf25e84af7832884004d8d5cb7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:15:50.079330 13032 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e0fceaf25e84af7832884004d8d5cb7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:15:50.079493 13032 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e0fceaf25e84af7832884004d8d5cb7 [term 1 LEADER]: Becoming Leader. State: Replica: 4e0fceaf25e84af7832884004d8d5cb7, State: Running, Role: LEADER
I20240501 14:15:50.079762 13032 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e0fceaf25e84af7832884004d8d5cb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e0fceaf25e84af7832884004d8d5cb7" member_type: VOTER last_known_addr { host: "127.7.39.190" port: 37333 } }
I20240501 14:15:50.079952 13027 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e0fceaf25e84af7832884004d8d5cb7 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:15:50.081123 13033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e0fceaf25e84af7832884004d8d5cb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e0fceaf25e84af7832884004d8d5cb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e0fceaf25e84af7832884004d8d5cb7" member_type: VOTER last_known_addr { host: "127.7.39.190" port: 37333 } } }
I20240501 14:15:50.081247 13033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e0fceaf25e84af7832884004d8d5cb7 [sys.catalog]: This master's current role is: LEADER
I20240501 14:15:50.081527 13034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e0fceaf25e84af7832884004d8d5cb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e0fceaf25e84af7832884004d8d5cb7. Latest consensus state: current_term: 1 leader_uuid: "4e0fceaf25e84af7832884004d8d5cb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e0fceaf25e84af7832884004d8d5cb7" member_type: VOTER last_known_addr { host: "127.7.39.190" port: 37333 } } }
I20240501 14:15:50.081605 13034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e0fceaf25e84af7832884004d8d5cb7 [sys.catalog]: This master's current role is: LEADER
W20240501 14:15:50.084128 13047 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4e0fceaf25e84af7832884004d8d5cb7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 14:15:50.084192 13047 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 14:15:50.084259 13041 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:15:50.084976 13041 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:15:50.089095 13041 catalog_manager.cc:1347] Generated new cluster ID: 4471fadb27d64a0f896f3a25f1c8f797
I20240501 14:15:50.089162 13041 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:15:50.104684 13041 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 14:15:50.105396 13041 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:15:50.116093 13041 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 4e0fceaf25e84af7832884004d8d5cb7: Generated new TSK 0
I20240501 14:15:50.116727 13041 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:15:50.125094 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.129:0
--local_ip_for_outbound_sockets=127.7.39.129
--webserver_interface=127.7.39.129
--webserver_port=0
--tserver_master_addrs=127.7.39.190:37333
--builtin_ntp_servers=127.7.39.148:33663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 14:15:50.231393 13051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:15:50.231592 13051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:15:50.231663 13051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:15:50.234549 13051 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:15:50.234611 13051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:15:50.234721 13051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.129
I20240501 14:15:50.238152 13051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:33663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.7.39.129
--webserver_port=0
--tserver_master_addrs=127.7.39.190:37333
--never_fsync=true
--heap_profile_path=/tmp/kudu.13051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.129
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:15:50.238893 13051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:15:50.240015 13051 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:15:50.247934 13059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:15:50.248454 13056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:15:50.249246 13057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:15:50.249775 13051 server_base.cc:1034] running on GCE node
I20240501 14:15:50.250108 13051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:15:50.250563 13051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:15:50.251716 13051 hybrid_clock.cc:648] HybridClock initialized: now 1714572950251707 us; error 23 us; skew 500 ppm
I20240501 14:15:50.253435 13051 webserver.cc:460] Webserver started at http://127.7.39.129:41729/ using document root <none> and password file <none>
I20240501 14:15:50.253911 13051 fs_manager.cc:359] Metadata directory not provided
I20240501 14:15:50.253968 13051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:15:50.254127 13051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:15:50.255643 13051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "85daee2113b445e293681247247f7f96"
format_stamp: "Formatted at 2024-05-01 14:15:50 on dist-test-slave-rnfw"
I20240501 14:15:50.257457 13051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "85daee2113b445e293681247247f7f96"
format_stamp: "Formatted at 2024-05-01 14:15:50 on dist-test-slave-rnfw"
I20240501 14:15:50.260681 13051 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:15:50.262658 13065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:15:50.263576 13051 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:15:50.263671 13051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "85daee2113b445e293681247247f7f96"
format_stamp: "Formatted at 2024-05-01 14:15:50 on dist-test-slave-rnfw"
I20240501 14:15:50.263751 13051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:15:50.281488 13051 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:15:50.281955 13051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:15:50.282110 13051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:15:50.282575 13051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:15:50.283393 13051 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:15:50.283447 13051 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:15:50.283507 13051 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:15:50.283540 13051 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:15:50.292315 13051 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.129:34901
I20240501 14:15:50.292333 13178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.129:34901 every 8 connection(s)
I20240501 14:15:50.293123 13051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 14:15:50.300643 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 13051
I20240501 14:15:50.300772 7326 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 14:15:50.300989 13179 heartbeater.cc:347] Connected to a master server at 127.7.39.190:37333
I20240501 14:15:50.301182 13179 heartbeater.cc:464] Registering TS with master...
I20240501 14:15:50.301647 13179 heartbeater.cc:511] Master 127.7.39.190:37333 requested a full tablet report, sending...
I20240501 14:15:50.302691 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.130:0
--local_ip_for_outbound_sockets=127.7.39.130
--webserver_interface=127.7.39.130
--webserver_port=0
--tserver_master_addrs=127.7.39.190:37333
--builtin_ntp_servers=127.7.39.148:33663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:15:50.302909 12991 ts_manager.cc:194] Registered new tserver with Master: 85daee2113b445e293681247247f7f96 (127.7.39.129:34901)
I20240501 14:15:50.304267 12991 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.129:54353
W20240501 14:15:50.414801 13182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:15:50.415006 13182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:15:50.415060 13182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:15:50.417737 13182 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:15:50.417791 13182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:15:50.417850 13182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.130
I20240501 14:15:50.421186 13182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:33663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.7.39.130
--webserver_port=0
--tserver_master_addrs=127.7.39.190:37333
--never_fsync=true
--heap_profile_path=/tmp/kudu.13182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.130
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:15:50.421912 13182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:15:50.422850 13182 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:15:50.430859 13188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:15:50.431315 13190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:15:50.432006 13187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:15:50.432641 13182 server_base.cc:1034] running on GCE node
I20240501 14:15:50.432977 13182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:15:50.433449 13182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:15:50.434597 13182 hybrid_clock.cc:648] HybridClock initialized: now 1714572950434588 us; error 24 us; skew 500 ppm
I20240501 14:15:50.436499 13182 webserver.cc:460] Webserver started at http://127.7.39.130:33465/ using document root <none> and password file <none>
I20240501 14:15:50.436980 13182 fs_manager.cc:359] Metadata directory not provided
I20240501 14:15:50.437031 13182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:15:50.437178 13182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:15:50.438664 13182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "79c63483422d4228a8f1b74b871f426f"
format_stamp: "Formatted at 2024-05-01 14:15:50 on dist-test-slave-rnfw"
I20240501 14:15:50.439157 13182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "79c63483422d4228a8f1b74b871f426f"
format_stamp: "Formatted at 2024-05-01 14:15:50 on dist-test-slave-rnfw"
I20240501 14:15:50.442204 13182 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20240501 14:15:50.444229 13196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:15:50.445155 13182 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 14:15:50.445243 13182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "79c63483422d4228a8f1b74b871f426f"
format_stamp: "Formatted at 2024-05-01 14:15:50 on dist-test-slave-rnfw"
I20240501 14:15:50.445324 13182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:15:50.464212 13182 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:15:50.464668 13182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:15:50.464812 13182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:15:50.465234 13182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:15:50.465991 13182 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:15:50.466038 13182 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:15:50.466077 13182 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:15:50.466109 13182 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:15:50.474040 13182 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.130:44935
I20240501 14:15:50.474071 13309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.130:44935 every 8 connection(s)
I20240501 14:15:50.474825 13182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 14:15:50.479838 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 13182
I20240501 14:15:50.479940 7326 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 14:15:50.482232 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.131:0
--local_ip_for_outbound_sockets=127.7.39.131
--webserver_interface=127.7.39.131
--webserver_port=0
--tserver_master_addrs=127.7.39.190:37333
--builtin_ntp_servers=127.7.39.148:33663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:15:50.482848 13310 heartbeater.cc:347] Connected to a master server at 127.7.39.190:37333
I20240501 14:15:50.483093 13310 heartbeater.cc:464] Registering TS with master...
I20240501 14:15:50.483556 13310 heartbeater.cc:511] Master 127.7.39.190:37333 requested a full tablet report, sending...
I20240501 14:15:50.484519 12991 ts_manager.cc:194] Registered new tserver with Master: 79c63483422d4228a8f1b74b871f426f (127.7.39.130:44935)
I20240501 14:15:50.485018 12991 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.130:34293
W20240501 14:15:50.580413 13313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:15:50.580585 13313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:15:50.580636 13313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:15:50.583622 13313 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:15:50.583678 13313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:15:50.583757 13313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.131
I20240501 14:15:50.586997 13313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:33663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.7.39.131
--webserver_port=0
--tserver_master_addrs=127.7.39.190:37333
--never_fsync=true
--heap_profile_path=/tmp/kudu.13313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.131
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:15:50.587755 13313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:15:50.588716 13313 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:15:50.596297 13318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:15:50.597024 13321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:15:50.597437 13319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:15:50.598244 13313 server_base.cc:1034] running on GCE node
I20240501 14:15:50.598551 13313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:15:50.599016 13313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:15:50.600216 13313 hybrid_clock.cc:648] HybridClock initialized: now 1714572950600205 us; error 27 us; skew 500 ppm
I20240501 14:15:50.601926 13313 webserver.cc:460] Webserver started at http://127.7.39.131:34521/ using document root <none> and password file <none>
I20240501 14:15:50.602419 13313 fs_manager.cc:359] Metadata directory not provided
I20240501 14:15:50.602474 13313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:15:50.602646 13313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:15:50.604215 13313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "76cb2fb202294a2d87080b62ca189bb9"
format_stamp: "Formatted at 2024-05-01 14:15:50 on dist-test-slave-rnfw"
I20240501 14:15:50.604671 13313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "76cb2fb202294a2d87080b62ca189bb9"
format_stamp: "Formatted at 2024-05-01 14:15:50 on dist-test-slave-rnfw"
I20240501 14:15:50.607761 13313 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:15:50.609722 13327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:15:50.610616 13313 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:15:50.610702 13313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "76cb2fb202294a2d87080b62ca189bb9"
format_stamp: "Formatted at 2024-05-01 14:15:50 on dist-test-slave-rnfw"
I20240501 14:15:50.610795 13313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:15:50.631765 13313 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:15:50.632248 13313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:15:50.632416 13313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:15:50.632871 13313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:15:50.633637 13313 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:15:50.633690 13313 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:15:50.633745 13313 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:15:50.633773 13313 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:15:50.641630 13313 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.131:33077
I20240501 14:15:50.641670 13440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.131:33077 every 8 connection(s)
I20240501 14:15:50.642490 13313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 14:15:50.648810 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 13313
I20240501 14:15:50.648924 7326 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 14:15:50.650506 13441 heartbeater.cc:347] Connected to a master server at 127.7.39.190:37333
I20240501 14:15:50.650682 13441 heartbeater.cc:464] Registering TS with master...
I20240501 14:15:50.650931 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.132:0
--local_ip_for_outbound_sockets=127.7.39.132
--webserver_interface=127.7.39.132
--webserver_port=0
--tserver_master_addrs=127.7.39.190:37333
--builtin_ntp_servers=127.7.39.148:33663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:15:50.651124 13441 heartbeater.cc:511] Master 127.7.39.190:37333 requested a full tablet report, sending...
I20240501 14:15:50.651973 12991 ts_manager.cc:194] Registered new tserver with Master: 76cb2fb202294a2d87080b62ca189bb9 (127.7.39.131:33077)
I20240501 14:15:50.652462 12991 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.131:45749
W20240501 14:15:50.749230 13444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:15:50.749428 13444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:15:50.749477 13444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:15:50.752346 13444 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:15:50.752403 13444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:15:50.752488 13444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.132
I20240501 14:15:50.755736 13444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:33663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.7.39.132
--webserver_port=0
--tserver_master_addrs=127.7.39.190:37333
--never_fsync=true
--heap_profile_path=/tmp/kudu.13444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.132
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:15:50.756418 13444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:15:50.757361 13444 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:15:50.765029 13449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:15:50.765704 13450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:15:50.766259 13452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:15:50.766963 13444 server_base.cc:1034] running on GCE node
I20240501 14:15:50.767308 13444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:15:50.767769 13444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:15:50.768923 13444 hybrid_clock.cc:648] HybridClock initialized: now 1714572950768899 us; error 37 us; skew 500 ppm
I20240501 14:15:50.770567 13444 webserver.cc:460] Webserver started at http://127.7.39.132:33535/ using document root <none> and password file <none>
I20240501 14:15:50.771067 13444 fs_manager.cc:359] Metadata directory not provided
I20240501 14:15:50.771133 13444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:15:50.771306 13444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:15:50.772778 13444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "9b9e9b6a68404d638f1edeeefe7c2633"
format_stamp: "Formatted at 2024-05-01 14:15:50 on dist-test-slave-rnfw"
I20240501 14:15:50.773213 13444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "9b9e9b6a68404d638f1edeeefe7c2633"
format_stamp: "Formatted at 2024-05-01 14:15:50 on dist-test-slave-rnfw"
I20240501 14:15:50.776084 13444 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:15:50.777930 13458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:15:50.778766 13444 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:15:50.778864 13444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "9b9e9b6a68404d638f1edeeefe7c2633"
format_stamp: "Formatted at 2024-05-01 14:15:50 on dist-test-slave-rnfw"
I20240501 14:15:50.778939 13444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:15:50.802567 13444 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:15:50.803041 13444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:15:50.803223 13444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:15:50.803670 13444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:15:50.804419 13444 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:15:50.804466 13444 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:15:50.804517 13444 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:15:50.804540 13444 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:15:50.812764 13444 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.132:39407
I20240501 14:15:50.812793 13571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.132:39407 every 8 connection(s)
I20240501 14:15:50.813637 13444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 14:15:50.817135 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 13444
I20240501 14:15:50.817253 7326 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 14:15:50.821252 13572 heartbeater.cc:347] Connected to a master server at 127.7.39.190:37333
I20240501 14:15:50.821426 13572 heartbeater.cc:464] Registering TS with master...
I20240501 14:15:50.821818 13572 heartbeater.cc:511] Master 127.7.39.190:37333 requested a full tablet report, sending...
I20240501 14:15:50.822615 12991 ts_manager.cc:194] Registered new tserver with Master: 9b9e9b6a68404d638f1edeeefe7c2633 (127.7.39.132:39407)
I20240501 14:15:50.823172 12991 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.132:34313
I20240501 14:15:50.830755 7326 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240501 14:15:50.838595 12991 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50778:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 14:15:50.852991 13113 tablet_service.cc:1467] Processing CreateTablet for tablet b02f666697f34d15a1ae1a38c0e44cd6 (DEFAULT_TABLE table=TestTable [id=75aa898f623346b5b29cc80eeb9ef506]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:15:50.853803 13113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b02f666697f34d15a1ae1a38c0e44cd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:15:50.858570 13585 tablet_bootstrap.cc:492] T b02f666697f34d15a1ae1a38c0e44cd6 P 85daee2113b445e293681247247f7f96: Bootstrap starting.
I20240501 14:15:50.860414 13585 tablet_bootstrap.cc:654] T b02f666697f34d15a1ae1a38c0e44cd6 P 85daee2113b445e293681247247f7f96: Neither blocks nor log segments found. Creating new log.
I20240501 14:15:50.861006 13585 log.cc:824] T b02f666697f34d15a1ae1a38c0e44cd6 P 85daee2113b445e293681247247f7f96: Log is configured to *not* fsync() on all Append() calls
I20240501 14:15:50.862394 13585 tablet_bootstrap.cc:492] T b02f666697f34d15a1ae1a38c0e44cd6 P 85daee2113b445e293681247247f7f96: No bootstrap required, opened a new log
I20240501 14:15:50.862555 13585 ts_tablet_manager.cc:1367] T b02f666697f34d15a1ae1a38c0e44cd6 P 85daee2113b445e293681247247f7f96: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:15:50.865164 13585 raft_consensus.cc:348] T b02f666697f34d15a1ae1a38c0e44cd6 P 85daee2113b445e293681247247f7f96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85daee2113b445e293681247247f7f96" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 34901 } }
I20240501 14:15:50.865309 13585 raft_consensus.cc:374] T b02f666697f34d15a1ae1a38c0e44cd6 P 85daee2113b445e293681247247f7f96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:15:50.865348 13585 raft_consensus.cc:729] T b02f666697f34d15a1ae1a38c0e44cd6 P 85daee2113b445e293681247247f7f96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85daee2113b445e293681247247f7f96, State: Initialized, Role: FOLLOWER
I20240501 14:15:50.865670 13585 consensus_queue.cc:260] T b02f666697f34d15a1ae1a38c0e44cd6 P 85daee2113b445e293681247247f7f96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85daee2113b445e293681247247f7f96" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 34901 } }
I20240501 14:15:50.865767 13585 raft_consensus.cc:388] T b02f666697f34d15a1ae1a38c0e44cd6 P 85daee2113b445e293681247247f7f96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:15:50.865801 13585 raft_consensus.cc:482] T b02f666697f34d15a1ae1a38c0e44cd6 P 85daee2113b445e293681247247f7f96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:15:50.865870 13585 raft_consensus.cc:3037] T b02f666697f34d15a1ae1a38c0e44cd6 P 85daee2113b445e293681247247f7f96 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:15:50.866631 13585 raft_consensus.cc:504] T b02f666697f34d15a1ae1a38c0e44cd6 P 85daee2113b445e293681247247f7f96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85daee2113b445e293681247247f7f96" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 34901 } }
I20240501 14:15:50.866920 13585 leader_election.cc:302] T b02f666697f34d15a1ae1a38c0e44cd6 P 85daee2113b445e293681247247f7f96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85daee2113b445e293681247247f7f96; no voters:
I20240501 14:15:50.867228 13585 leader_election.cc:288] T b02f666697f34d15a1ae1a38c0e44cd6 P 85daee2113b445e293681247247f7f96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:15:50.867313 13587 raft_consensus.cc:2781] T b02f666697f34d15a1ae1a38c0e44cd6 P 85daee2113b445e293681247247f7f96 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:15:50.867532 13587 raft_consensus.cc:686] T b02f666697f34d15a1ae1a38c0e44cd6 P 85daee2113b445e293681247247f7f96 [term 1 LEADER]: Becoming Leader. State: Replica: 85daee2113b445e293681247247f7f96, State: Running, Role: LEADER
I20240501 14:15:50.867802 13179 heartbeater.cc:503] Master 127.7.39.190:37333 was elected leader, sending a full tablet report...
I20240501 14:15:50.867870 13587 consensus_queue.cc:237] T b02f666697f34d15a1ae1a38c0e44cd6 P 85daee2113b445e293681247247f7f96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85daee2113b445e293681247247f7f96" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 34901 } }
I20240501 14:15:50.867962 13585 ts_tablet_manager.cc:1398] T b02f666697f34d15a1ae1a38c0e44cd6 P 85daee2113b445e293681247247f7f96: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240501 14:15:50.870473 12991 catalog_manager.cc:5484] T b02f666697f34d15a1ae1a38c0e44cd6 P 85daee2113b445e293681247247f7f96 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85daee2113b445e293681247247f7f96 (127.7.39.129). New cstate: current_term: 1 leader_uuid: "85daee2113b445e293681247247f7f96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85daee2113b445e293681247247f7f96" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 34901 } } }
I20240501 14:15:50.879057 7326 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240501 14:15:50.880295 7326 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 85daee2113b445e293681247247f7f96 to finish bootstrapping
I20240501 14:15:50.885354 7326 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 79c63483422d4228a8f1b74b871f426f to finish bootstrapping
I20240501 14:15:50.890575 7326 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 76cb2fb202294a2d87080b62ca189bb9 to finish bootstrapping
I20240501 14:15:50.896131 7326 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 9b9e9b6a68404d638f1edeeefe7c2633 to finish bootstrapping
I20240501 14:15:50.900803 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 13313
I20240501 14:15:50.906510 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 13444
I20240501 14:15:51.486982 13310 heartbeater.cc:503] Master 127.7.39.190:37333 was elected leader, sending a full tablet report...
I20240501 14:15:54.663185 12991 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50778:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:15:54.669953 13113 tablet_service.cc:1467] Processing CreateTablet for tablet 022b6a678a1b46a8b8fe6aae03cda95f (DEFAULT_TABLE table=rebalance_test_table_0 [id=f47200dc525f4d20bf24e56c9b6523e6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:15:54.670284 13113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 022b6a678a1b46a8b8fe6aae03cda95f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:15:54.672510 13602 tablet_bootstrap.cc:492] T 022b6a678a1b46a8b8fe6aae03cda95f P 85daee2113b445e293681247247f7f96: Bootstrap starting.
I20240501 14:15:54.673347 13602 tablet_bootstrap.cc:654] T 022b6a678a1b46a8b8fe6aae03cda95f P 85daee2113b445e293681247247f7f96: Neither blocks nor log segments found. Creating new log.
I20240501 14:15:54.674346 13602 tablet_bootstrap.cc:492] T 022b6a678a1b46a8b8fe6aae03cda95f P 85daee2113b445e293681247247f7f96: No bootstrap required, opened a new log
I20240501 14:15:54.674419 13602 ts_tablet_manager.cc:1367] T 022b6a678a1b46a8b8fe6aae03cda95f P 85daee2113b445e293681247247f7f96: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:15:54.674914 13602 raft_consensus.cc:348] T 022b6a678a1b46a8b8fe6aae03cda95f P 85daee2113b445e293681247247f7f96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85daee2113b445e293681247247f7f96" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 34901 } }
I20240501 14:15:54.675006 13602 raft_consensus.cc:374] T 022b6a678a1b46a8b8fe6aae03cda95f P 85daee2113b445e293681247247f7f96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:15:54.674719 13243 tablet_service.cc:1467] Processing CreateTablet for tablet 3a40e3962fa442d49b12592b9761a73d (DEFAULT_TABLE table=rebalance_test_table_0 [id=f47200dc525f4d20bf24e56c9b6523e6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:15:54.675056 13602 raft_consensus.cc:729] T 022b6a678a1b46a8b8fe6aae03cda95f P 85daee2113b445e293681247247f7f96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85daee2113b445e293681247247f7f96, State: Initialized, Role: FOLLOWER
I20240501 14:15:54.675201 13602 consensus_queue.cc:260] T 022b6a678a1b46a8b8fe6aae03cda95f P 85daee2113b445e293681247247f7f96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85daee2113b445e293681247247f7f96" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 34901 } }
I20240501 14:15:54.675282 13602 raft_consensus.cc:388] T 022b6a678a1b46a8b8fe6aae03cda95f P 85daee2113b445e293681247247f7f96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:15:54.675311 13602 raft_consensus.cc:482] T 022b6a678a1b46a8b8fe6aae03cda95f P 85daee2113b445e293681247247f7f96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:15:54.675355 13602 raft_consensus.cc:3037] T 022b6a678a1b46a8b8fe6aae03cda95f P 85daee2113b445e293681247247f7f96 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:15:54.675619 13243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a40e3962fa442d49b12592b9761a73d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:15:54.676129 13602 raft_consensus.cc:504] T 022b6a678a1b46a8b8fe6aae03cda95f P 85daee2113b445e293681247247f7f96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85daee2113b445e293681247247f7f96" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 34901 } }
I20240501 14:15:54.676254 13602 leader_election.cc:302] T 022b6a678a1b46a8b8fe6aae03cda95f P 85daee2113b445e293681247247f7f96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85daee2113b445e293681247247f7f96; no voters:
I20240501 14:15:54.676409 13602 leader_election.cc:288] T 022b6a678a1b46a8b8fe6aae03cda95f P 85daee2113b445e293681247247f7f96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:15:54.676501 13604 raft_consensus.cc:2781] T 022b6a678a1b46a8b8fe6aae03cda95f P 85daee2113b445e293681247247f7f96 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:15:54.676663 13602 ts_tablet_manager.cc:1398] T 022b6a678a1b46a8b8fe6aae03cda95f P 85daee2113b445e293681247247f7f96: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 14:15:54.676695 13604 raft_consensus.cc:686] T 022b6a678a1b46a8b8fe6aae03cda95f P 85daee2113b445e293681247247f7f96 [term 1 LEADER]: Becoming Leader. State: Replica: 85daee2113b445e293681247247f7f96, State: Running, Role: LEADER
I20240501 14:15:54.676803 13604 consensus_queue.cc:237] T 022b6a678a1b46a8b8fe6aae03cda95f P 85daee2113b445e293681247247f7f96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85daee2113b445e293681247247f7f96" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 34901 } }
I20240501 14:15:54.677870 12991 catalog_manager.cc:5484] T 022b6a678a1b46a8b8fe6aae03cda95f P 85daee2113b445e293681247247f7f96 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85daee2113b445e293681247247f7f96 (127.7.39.129). New cstate: current_term: 1 leader_uuid: "85daee2113b445e293681247247f7f96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85daee2113b445e293681247247f7f96" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 34901 } } }
I20240501 14:15:54.674711 13244 tablet_service.cc:1467] Processing CreateTablet for tablet a9807ddd18aa48e69bcfcc7def51d1b0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f47200dc525f4d20bf24e56c9b6523e6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:15:54.679255 13244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9807ddd18aa48e69bcfcc7def51d1b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:15:54.680609 13609 tablet_bootstrap.cc:492] T 3a40e3962fa442d49b12592b9761a73d P 79c63483422d4228a8f1b74b871f426f: Bootstrap starting.
I20240501 14:15:54.682408 13609 tablet_bootstrap.cc:654] T 3a40e3962fa442d49b12592b9761a73d P 79c63483422d4228a8f1b74b871f426f: Neither blocks nor log segments found. Creating new log.
I20240501 14:15:54.683075 13609 log.cc:824] T 3a40e3962fa442d49b12592b9761a73d P 79c63483422d4228a8f1b74b871f426f: Log is configured to *not* fsync() on all Append() calls
I20240501 14:15:54.684275 13609 tablet_bootstrap.cc:492] T 3a40e3962fa442d49b12592b9761a73d P 79c63483422d4228a8f1b74b871f426f: No bootstrap required, opened a new log
I20240501 14:15:54.684414 13609 ts_tablet_manager.cc:1367] T 3a40e3962fa442d49b12592b9761a73d P 79c63483422d4228a8f1b74b871f426f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:15:54.686872 13609 raft_consensus.cc:348] T 3a40e3962fa442d49b12592b9761a73d P 79c63483422d4228a8f1b74b871f426f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c63483422d4228a8f1b74b871f426f" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 44935 } }
I20240501 14:15:54.687027 13609 raft_consensus.cc:374] T 3a40e3962fa442d49b12592b9761a73d P 79c63483422d4228a8f1b74b871f426f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:15:54.687104 13609 raft_consensus.cc:729] T 3a40e3962fa442d49b12592b9761a73d P 79c63483422d4228a8f1b74b871f426f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79c63483422d4228a8f1b74b871f426f, State: Initialized, Role: FOLLOWER
I20240501 14:15:54.687472 13609 consensus_queue.cc:260] T 3a40e3962fa442d49b12592b9761a73d P 79c63483422d4228a8f1b74b871f426f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c63483422d4228a8f1b74b871f426f" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 44935 } }
I20240501 14:15:54.687577 13609 raft_consensus.cc:388] T 3a40e3962fa442d49b12592b9761a73d P 79c63483422d4228a8f1b74b871f426f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:15:54.687630 13609 raft_consensus.cc:482] T 3a40e3962fa442d49b12592b9761a73d P 79c63483422d4228a8f1b74b871f426f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:15:54.687721 13609 raft_consensus.cc:3037] T 3a40e3962fa442d49b12592b9761a73d P 79c63483422d4228a8f1b74b871f426f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:15:54.688328 13609 raft_consensus.cc:504] T 3a40e3962fa442d49b12592b9761a73d P 79c63483422d4228a8f1b74b871f426f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c63483422d4228a8f1b74b871f426f" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 44935 } }
I20240501 14:15:54.688580 13609 leader_election.cc:302] T 3a40e3962fa442d49b12592b9761a73d P 79c63483422d4228a8f1b74b871f426f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79c63483422d4228a8f1b74b871f426f; no voters:
I20240501 14:15:54.688802 13609 leader_election.cc:288] T 3a40e3962fa442d49b12592b9761a73d P 79c63483422d4228a8f1b74b871f426f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:15:54.688871 13611 raft_consensus.cc:2781] T 3a40e3962fa442d49b12592b9761a73d P 79c63483422d4228a8f1b74b871f426f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:15:54.689055 13611 raft_consensus.cc:686] T 3a40e3962fa442d49b12592b9761a73d P 79c63483422d4228a8f1b74b871f426f [term 1 LEADER]: Becoming Leader. State: Replica: 79c63483422d4228a8f1b74b871f426f, State: Running, Role: LEADER
I20240501 14:15:54.689296 13611 consensus_queue.cc:237] T 3a40e3962fa442d49b12592b9761a73d P 79c63483422d4228a8f1b74b871f426f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c63483422d4228a8f1b74b871f426f" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 44935 } }
I20240501 14:15:54.689702 13609 ts_tablet_manager.cc:1398] T 3a40e3962fa442d49b12592b9761a73d P 79c63483422d4228a8f1b74b871f426f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240501 14:15:54.689882 13609 tablet_bootstrap.cc:492] T a9807ddd18aa48e69bcfcc7def51d1b0 P 79c63483422d4228a8f1b74b871f426f: Bootstrap starting.
I20240501 14:15:54.690687 13609 tablet_bootstrap.cc:654] T a9807ddd18aa48e69bcfcc7def51d1b0 P 79c63483422d4228a8f1b74b871f426f: Neither blocks nor log segments found. Creating new log.
I20240501 14:15:54.691071 12991 catalog_manager.cc:5484] T 3a40e3962fa442d49b12592b9761a73d P 79c63483422d4228a8f1b74b871f426f reported cstate change: term changed from 0 to 1, leader changed from <none> to 79c63483422d4228a8f1b74b871f426f (127.7.39.130). New cstate: current_term: 1 leader_uuid: "79c63483422d4228a8f1b74b871f426f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c63483422d4228a8f1b74b871f426f" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 44935 } } }
I20240501 14:15:54.691648 13609 tablet_bootstrap.cc:492] T a9807ddd18aa48e69bcfcc7def51d1b0 P 79c63483422d4228a8f1b74b871f426f: No bootstrap required, opened a new log
I20240501 14:15:54.691727 13609 ts_tablet_manager.cc:1367] T a9807ddd18aa48e69bcfcc7def51d1b0 P 79c63483422d4228a8f1b74b871f426f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:15:54.692198 13609 raft_consensus.cc:348] T a9807ddd18aa48e69bcfcc7def51d1b0 P 79c63483422d4228a8f1b74b871f426f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c63483422d4228a8f1b74b871f426f" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 44935 } }
I20240501 14:15:54.692287 13609 raft_consensus.cc:374] T a9807ddd18aa48e69bcfcc7def51d1b0 P 79c63483422d4228a8f1b74b871f426f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:15:54.692319 13609 raft_consensus.cc:729] T a9807ddd18aa48e69bcfcc7def51d1b0 P 79c63483422d4228a8f1b74b871f426f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79c63483422d4228a8f1b74b871f426f, State: Initialized, Role: FOLLOWER
I20240501 14:15:54.692428 13609 consensus_queue.cc:260] T a9807ddd18aa48e69bcfcc7def51d1b0 P 79c63483422d4228a8f1b74b871f426f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c63483422d4228a8f1b74b871f426f" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 44935 } }
I20240501 14:15:54.692494 13609 raft_consensus.cc:388] T a9807ddd18aa48e69bcfcc7def51d1b0 P 79c63483422d4228a8f1b74b871f426f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:15:54.692538 13609 raft_consensus.cc:482] T a9807ddd18aa48e69bcfcc7def51d1b0 P 79c63483422d4228a8f1b74b871f426f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:15:54.692585 13609 raft_consensus.cc:3037] T a9807ddd18aa48e69bcfcc7def51d1b0 P 79c63483422d4228a8f1b74b871f426f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:15:54.693243 13609 raft_consensus.cc:504] T a9807ddd18aa48e69bcfcc7def51d1b0 P 79c63483422d4228a8f1b74b871f426f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c63483422d4228a8f1b74b871f426f" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 44935 } }
I20240501 14:15:54.693356 13609 leader_election.cc:302] T a9807ddd18aa48e69bcfcc7def51d1b0 P 79c63483422d4228a8f1b74b871f426f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79c63483422d4228a8f1b74b871f426f; no voters:
I20240501 14:15:54.693440 13609 leader_election.cc:288] T a9807ddd18aa48e69bcfcc7def51d1b0 P 79c63483422d4228a8f1b74b871f426f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:15:54.693490 13611 raft_consensus.cc:2781] T a9807ddd18aa48e69bcfcc7def51d1b0 P 79c63483422d4228a8f1b74b871f426f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:15:54.693578 13611 raft_consensus.cc:686] T a9807ddd18aa48e69bcfcc7def51d1b0 P 79c63483422d4228a8f1b74b871f426f [term 1 LEADER]: Becoming Leader. State: Replica: 79c63483422d4228a8f1b74b871f426f, State: Running, Role: LEADER
I20240501 14:15:54.693622 13609 ts_tablet_manager.cc:1398] T a9807ddd18aa48e69bcfcc7def51d1b0 P 79c63483422d4228a8f1b74b871f426f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:15:54.693696 13611 consensus_queue.cc:237] T a9807ddd18aa48e69bcfcc7def51d1b0 P 79c63483422d4228a8f1b74b871f426f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c63483422d4228a8f1b74b871f426f" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 44935 } }
I20240501 14:15:54.694940 12991 catalog_manager.cc:5484] T a9807ddd18aa48e69bcfcc7def51d1b0 P 79c63483422d4228a8f1b74b871f426f reported cstate change: term changed from 0 to 1, leader changed from <none> to 79c63483422d4228a8f1b74b871f426f (127.7.39.130). New cstate: current_term: 1 leader_uuid: "79c63483422d4228a8f1b74b871f426f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c63483422d4228a8f1b74b871f426f" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 44935 } } }
I20240501 14:15:54.848145 12991 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50778:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:15:54.853205 13244 tablet_service.cc:1467] Processing CreateTablet for tablet fac046d9307f4f6ab55169083f3c0211 (DEFAULT_TABLE table=rebalance_test_table_1 [id=11bceb159f3a4fb58e08d418a7438d1d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:15:54.853355 13112 tablet_service.cc:1467] Processing CreateTablet for tablet cf57e7aaa70a43afb0bdfd0eb37f0709 (DEFAULT_TABLE table=rebalance_test_table_1 [id=11bceb159f3a4fb58e08d418a7438d1d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:15:54.853525 13244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fac046d9307f4f6ab55169083f3c0211. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:15:54.853682 13112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf57e7aaa70a43afb0bdfd0eb37f0709. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:15:54.855075 13609 tablet_bootstrap.cc:492] T fac046d9307f4f6ab55169083f3c0211 P 79c63483422d4228a8f1b74b871f426f: Bootstrap starting.
I20240501 14:15:54.855260 13113 tablet_service.cc:1467] Processing CreateTablet for tablet a94adcc02f9b4de8a20bb59c4d291485 (DEFAULT_TABLE table=rebalance_test_table_1 [id=11bceb159f3a4fb58e08d418a7438d1d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:15:54.855304 13602 tablet_bootstrap.cc:492] T cf57e7aaa70a43afb0bdfd0eb37f0709 P 85daee2113b445e293681247247f7f96: Bootstrap starting.
I20240501 14:15:54.855463 13113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a94adcc02f9b4de8a20bb59c4d291485. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:15:54.855916 13609 tablet_bootstrap.cc:654] T fac046d9307f4f6ab55169083f3c0211 P 79c63483422d4228a8f1b74b871f426f: Neither blocks nor log segments found. Creating new log.
I20240501 14:15:54.856012 13602 tablet_bootstrap.cc:654] T cf57e7aaa70a43afb0bdfd0eb37f0709 P 85daee2113b445e293681247247f7f96: Neither blocks nor log segments found. Creating new log.
I20240501 14:15:54.856901 13609 tablet_bootstrap.cc:492] T fac046d9307f4f6ab55169083f3c0211 P 79c63483422d4228a8f1b74b871f426f: No bootstrap required, opened a new log
I20240501 14:15:54.856994 13609 ts_tablet_manager.cc:1367] T fac046d9307f4f6ab55169083f3c0211 P 79c63483422d4228a8f1b74b871f426f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:15:54.857465 13609 raft_consensus.cc:348] T fac046d9307f4f6ab55169083f3c0211 P 79c63483422d4228a8f1b74b871f426f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c63483422d4228a8f1b74b871f426f" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 44935 } }
I20240501 14:15:54.857547 13609 raft_consensus.cc:374] T fac046d9307f4f6ab55169083f3c0211 P 79c63483422d4228a8f1b74b871f426f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:15:54.857578 13609 raft_consensus.cc:729] T fac046d9307f4f6ab55169083f3c0211 P 79c63483422d4228a8f1b74b871f426f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79c63483422d4228a8f1b74b871f426f, State: Initialized, Role: FOLLOWER
I20240501 14:15:54.857692 13609 consensus_queue.cc:260] T fac046d9307f4f6ab55169083f3c0211 P 79c63483422d4228a8f1b74b871f426f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c63483422d4228a8f1b74b871f426f" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 44935 } }
I20240501 14:15:54.857759 13609 raft_consensus.cc:388] T fac046d9307f4f6ab55169083f3c0211 P 79c63483422d4228a8f1b74b871f426f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:15:54.857800 13609 raft_consensus.cc:482] T fac046d9307f4f6ab55169083f3c0211 P 79c63483422d4228a8f1b74b871f426f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:15:54.857854 13609 raft_consensus.cc:3037] T fac046d9307f4f6ab55169083f3c0211 P 79c63483422d4228a8f1b74b871f426f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:15:54.858444 13609 raft_consensus.cc:504] T fac046d9307f4f6ab55169083f3c0211 P 79c63483422d4228a8f1b74b871f426f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c63483422d4228a8f1b74b871f426f" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 44935 } }
I20240501 14:15:54.858558 13609 leader_election.cc:302] T fac046d9307f4f6ab55169083f3c0211 P 79c63483422d4228a8f1b74b871f426f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79c63483422d4228a8f1b74b871f426f; no voters:
I20240501 14:15:54.858644 13609 leader_election.cc:288] T fac046d9307f4f6ab55169083f3c0211 P 79c63483422d4228a8f1b74b871f426f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:15:54.858712 13611 raft_consensus.cc:2781] T fac046d9307f4f6ab55169083f3c0211 P 79c63483422d4228a8f1b74b871f426f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:15:54.859014 13602 tablet_bootstrap.cc:492] T cf57e7aaa70a43afb0bdfd0eb37f0709 P 85daee2113b445e293681247247f7f96: No bootstrap required, opened a new log
I20240501 14:15:54.859145 13602 ts_tablet_manager.cc:1367] T cf57e7aaa70a43afb0bdfd0eb37f0709 P 85daee2113b445e293681247247f7f96: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:15:54.859548 13602 raft_consensus.cc:348] T cf57e7aaa70a43afb0bdfd0eb37f0709 P 85daee2113b445e293681247247f7f96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85daee2113b445e293681247247f7f96" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 34901 } }
I20240501 14:15:54.859642 13602 raft_consensus.cc:374] T cf57e7aaa70a43afb0bdfd0eb37f0709 P 85daee2113b445e293681247247f7f96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:15:54.859697 13602 raft_consensus.cc:729] T cf57e7aaa70a43afb0bdfd0eb37f0709 P 85daee2113b445e293681247247f7f96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85daee2113b445e293681247247f7f96, State: Initialized, Role: FOLLOWER
I20240501 14:15:54.859827 13602 consensus_queue.cc:260] T cf57e7aaa70a43afb0bdfd0eb37f0709 P 85daee2113b445e293681247247f7f96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85daee2113b445e293681247247f7f96" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 34901 } }
I20240501 14:15:54.859884 13602 raft_consensus.cc:388] T cf57e7aaa70a43afb0bdfd0eb37f0709 P 85daee2113b445e293681247247f7f96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:15:54.859925 13602 raft_consensus.cc:482] T cf57e7aaa70a43afb0bdfd0eb37f0709 P 85daee2113b445e293681247247f7f96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:15:54.859977 13602 raft_consensus.cc:3037] T cf57e7aaa70a43afb0bdfd0eb37f0709 P 85daee2113b445e293681247247f7f96 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:15:54.860534 13602 raft_consensus.cc:504] T cf57e7aaa70a43afb0bdfd0eb37f0709 P 85daee2113b445e293681247247f7f96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85daee2113b445e293681247247f7f96" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 34901 } }
I20240501 14:15:54.860639 13602 leader_election.cc:302] T cf57e7aaa70a43afb0bdfd0eb37f0709 P 85daee2113b445e293681247247f7f96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85daee2113b445e293681247247f7f96; no voters:
I20240501 14:15:54.860715 13602 leader_election.cc:288] T cf57e7aaa70a43afb0bdfd0eb37f0709 P 85daee2113b445e293681247247f7f96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:15:54.860778 13606 raft_consensus.cc:2781] T cf57e7aaa70a43afb0bdfd0eb37f0709 P 85daee2113b445e293681247247f7f96 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:15:54.860852 13606 raft_consensus.cc:686] T cf57e7aaa70a43afb0bdfd0eb37f0709 P 85daee2113b445e293681247247f7f96 [term 1 LEADER]: Becoming Leader. State: Replica: 85daee2113b445e293681247247f7f96, State: Running, Role: LEADER
I20240501 14:15:54.860949 13606 consensus_queue.cc:237] T cf57e7aaa70a43afb0bdfd0eb37f0709 P 85daee2113b445e293681247247f7f96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85daee2113b445e293681247247f7f96" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 34901 } }
I20240501 14:15:54.861127 13611 raft_consensus.cc:686] T fac046d9307f4f6ab55169083f3c0211 P 79c63483422d4228a8f1b74b871f426f [term 1 LEADER]: Becoming Leader. State: Replica: 79c63483422d4228a8f1b74b871f426f, State: Running, Role: LEADER
I20240501 14:15:54.861258 13611 consensus_queue.cc:237] T fac046d9307f4f6ab55169083f3c0211 P 79c63483422d4228a8f1b74b871f426f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c63483422d4228a8f1b74b871f426f" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 44935 } }
I20240501 14:15:54.860992 13602 ts_tablet_manager.cc:1398] T cf57e7aaa70a43afb0bdfd0eb37f0709 P 85daee2113b445e293681247247f7f96: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:15:54.861672 13602 tablet_bootstrap.cc:492] T a94adcc02f9b4de8a20bb59c4d291485 P 85daee2113b445e293681247247f7f96: Bootstrap starting.
I20240501 14:15:54.862633 12991 catalog_manager.cc:5484] T cf57e7aaa70a43afb0bdfd0eb37f0709 P 85daee2113b445e293681247247f7f96 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85daee2113b445e293681247247f7f96 (127.7.39.129). New cstate: current_term: 1 leader_uuid: "85daee2113b445e293681247247f7f96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85daee2113b445e293681247247f7f96" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 34901 } } }
I20240501 14:15:54.862754 13602 tablet_bootstrap.cc:654] T a94adcc02f9b4de8a20bb59c4d291485 P 85daee2113b445e293681247247f7f96: Neither blocks nor log segments found. Creating new log.
I20240501 14:15:54.863364 13609 ts_tablet_manager.cc:1398] T fac046d9307f4f6ab55169083f3c0211 P 79c63483422d4228a8f1b74b871f426f: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:15:54.863673 12990 catalog_manager.cc:5484] T fac046d9307f4f6ab55169083f3c0211 P 79c63483422d4228a8f1b74b871f426f reported cstate change: term changed from 0 to 1, leader changed from <none> to 79c63483422d4228a8f1b74b871f426f (127.7.39.130). New cstate: current_term: 1 leader_uuid: "79c63483422d4228a8f1b74b871f426f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c63483422d4228a8f1b74b871f426f" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 44935 } } }
I20240501 14:15:54.864012 13602 tablet_bootstrap.cc:492] T a94adcc02f9b4de8a20bb59c4d291485 P 85daee2113b445e293681247247f7f96: No bootstrap required, opened a new log
I20240501 14:15:54.864073 13602 ts_tablet_manager.cc:1367] T a94adcc02f9b4de8a20bb59c4d291485 P 85daee2113b445e293681247247f7f96: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:15:54.864521 13602 raft_consensus.cc:348] T a94adcc02f9b4de8a20bb59c4d291485 P 85daee2113b445e293681247247f7f96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85daee2113b445e293681247247f7f96" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 34901 } }
I20240501 14:15:54.864609 13602 raft_consensus.cc:374] T a94adcc02f9b4de8a20bb59c4d291485 P 85daee2113b445e293681247247f7f96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:15:54.864651 13602 raft_consensus.cc:729] T a94adcc02f9b4de8a20bb59c4d291485 P 85daee2113b445e293681247247f7f96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85daee2113b445e293681247247f7f96, State: Initialized, Role: FOLLOWER
I20240501 14:15:54.864763 13602 consensus_queue.cc:260] T a94adcc02f9b4de8a20bb59c4d291485 P 85daee2113b445e293681247247f7f96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85daee2113b445e293681247247f7f96" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 34901 } }
I20240501 14:15:54.864833 13602 raft_consensus.cc:388] T a94adcc02f9b4de8a20bb59c4d291485 P 85daee2113b445e293681247247f7f96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:15:54.864873 13602 raft_consensus.cc:482] T a94adcc02f9b4de8a20bb59c4d291485 P 85daee2113b445e293681247247f7f96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:15:54.864917 13602 raft_consensus.cc:3037] T a94adcc02f9b4de8a20bb59c4d291485 P 85daee2113b445e293681247247f7f96 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:15:54.865694 13602 raft_consensus.cc:504] T a94adcc02f9b4de8a20bb59c4d291485 P 85daee2113b445e293681247247f7f96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85daee2113b445e293681247247f7f96" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 34901 } }
I20240501 14:15:54.865808 13602 leader_election.cc:302] T a94adcc02f9b4de8a20bb59c4d291485 P 85daee2113b445e293681247247f7f96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85daee2113b445e293681247247f7f96; no voters:
I20240501 14:15:54.865893 13602 leader_election.cc:288] T a94adcc02f9b4de8a20bb59c4d291485 P 85daee2113b445e293681247247f7f96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:15:54.865926 13606 raft_consensus.cc:2781] T a94adcc02f9b4de8a20bb59c4d291485 P 85daee2113b445e293681247247f7f96 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:15:54.865995 13606 raft_consensus.cc:686] T a94adcc02f9b4de8a20bb59c4d291485 P 85daee2113b445e293681247247f7f96 [term 1 LEADER]: Becoming Leader. State: Replica: 85daee2113b445e293681247247f7f96, State: Running, Role: LEADER
I20240501 14:15:54.866097 13602 ts_tablet_manager.cc:1398] T a94adcc02f9b4de8a20bb59c4d291485 P 85daee2113b445e293681247247f7f96: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:15:54.866091 13606 consensus_queue.cc:237] T a94adcc02f9b4de8a20bb59c4d291485 P 85daee2113b445e293681247247f7f96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85daee2113b445e293681247247f7f96" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 34901 } }
I20240501 14:15:54.867722 12991 catalog_manager.cc:5484] T a94adcc02f9b4de8a20bb59c4d291485 P 85daee2113b445e293681247247f7f96 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85daee2113b445e293681247247f7f96 (127.7.39.129). New cstate: current_term: 1 leader_uuid: "85daee2113b445e293681247247f7f96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85daee2113b445e293681247247f7f96" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 34901 } } }
I20240501 14:15:55.014775 12990 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50778:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:15:55.019647 13244 tablet_service.cc:1467] Processing CreateTablet for tablet 78c07f56e3dc4737bde64c7c9d8d55a2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6a94b2ce005e475a96c826c579584982]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:15:55.019731 13243 tablet_service.cc:1467] Processing CreateTablet for tablet edc2eeb4e0864195b6a709a525a48c54 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6a94b2ce005e475a96c826c579584982]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:15:55.019717 13113 tablet_service.cc:1467] Processing CreateTablet for tablet 23e203e290364f39ac4ed90570d5aba7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6a94b2ce005e475a96c826c579584982]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:15:55.019968 13244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78c07f56e3dc4737bde64c7c9d8d55a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:15:55.019969 13113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23e203e290364f39ac4ed90570d5aba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:15:55.020169 13243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edc2eeb4e0864195b6a709a525a48c54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:15:55.021557 13609 tablet_bootstrap.cc:492] T 78c07f56e3dc4737bde64c7c9d8d55a2 P 79c63483422d4228a8f1b74b871f426f: Bootstrap starting.
I20240501 14:15:55.021937 13602 tablet_bootstrap.cc:492] T 23e203e290364f39ac4ed90570d5aba7 P 85daee2113b445e293681247247f7f96: Bootstrap starting.
I20240501 14:15:55.022485 13609 tablet_bootstrap.cc:654] T 78c07f56e3dc4737bde64c7c9d8d55a2 P 79c63483422d4228a8f1b74b871f426f: Neither blocks nor log segments found. Creating new log.
I20240501 14:15:55.022862 13602 tablet_bootstrap.cc:654] T 23e203e290364f39ac4ed90570d5aba7 P 85daee2113b445e293681247247f7f96: Neither blocks nor log segments found. Creating new log.
I20240501 14:15:55.023592 13609 tablet_bootstrap.cc:492] T 78c07f56e3dc4737bde64c7c9d8d55a2 P 79c63483422d4228a8f1b74b871f426f: No bootstrap required, opened a new log
I20240501 14:15:55.023679 13609 ts_tablet_manager.cc:1367] T 78c07f56e3dc4737bde64c7c9d8d55a2 P 79c63483422d4228a8f1b74b871f426f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:15:55.023896 13602 tablet_bootstrap.cc:492] T 23e203e290364f39ac4ed90570d5aba7 P 85daee2113b445e293681247247f7f96: No bootstrap required, opened a new log
I20240501 14:15:55.023973 13602 ts_tablet_manager.cc:1367] T 23e203e290364f39ac4ed90570d5aba7 P 85daee2113b445e293681247247f7f96: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:15:55.024104 13609 raft_consensus.cc:348] T 78c07f56e3dc4737bde64c7c9d8d55a2 P 79c63483422d4228a8f1b74b871f426f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c63483422d4228a8f1b74b871f426f" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 44935 } }
I20240501 14:15:55.024199 13609 raft_consensus.cc:374] T 78c07f56e3dc4737bde64c7c9d8d55a2 P 79c63483422d4228a8f1b74b871f426f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:15:55.024241 13609 raft_consensus.cc:729] T 78c07f56e3dc4737bde64c7c9d8d55a2 P 79c63483422d4228a8f1b74b871f426f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79c63483422d4228a8f1b74b871f426f, State: Initialized, Role: FOLLOWER
I20240501 14:15:55.024348 13609 consensus_queue.cc:260] T 78c07f56e3dc4737bde64c7c9d8d55a2 P 79c63483422d4228a8f1b74b871f426f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c63483422d4228a8f1b74b871f426f" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 44935 } }
I20240501 14:15:55.024379 13602 raft_consensus.cc:348] T 23e203e290364f39ac4ed90570d5aba7 P 85daee2113b445e293681247247f7f96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85daee2113b445e293681247247f7f96" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 34901 } }
I20240501 14:15:55.024437 13609 raft_consensus.cc:388] T 78c07f56e3dc4737bde64c7c9d8d55a2 P 79c63483422d4228a8f1b74b871f426f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:15:55.024448 13602 raft_consensus.cc:374] T 23e203e290364f39ac4ed90570d5aba7 P 85daee2113b445e293681247247f7f96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:15:55.024482 13609 raft_consensus.cc:482] T 78c07f56e3dc4737bde64c7c9d8d55a2 P 79c63483422d4228a8f1b74b871f426f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:15:55.024482 13602 raft_consensus.cc:729] T 23e203e290364f39ac4ed90570d5aba7 P 85daee2113b445e293681247247f7f96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85daee2113b445e293681247247f7f96, State: Initialized, Role: FOLLOWER
I20240501 14:15:55.024530 13609 raft_consensus.cc:3037] T 78c07f56e3dc4737bde64c7c9d8d55a2 P 79c63483422d4228a8f1b74b871f426f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:15:55.024597 13602 consensus_queue.cc:260] T 23e203e290364f39ac4ed90570d5aba7 P 85daee2113b445e293681247247f7f96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85daee2113b445e293681247247f7f96" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 34901 } }
I20240501 14:15:55.024693 13602 raft_consensus.cc:388] T 23e203e290364f39ac4ed90570d5aba7 P 85daee2113b445e293681247247f7f96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:15:55.024744 13602 raft_consensus.cc:482] T 23e203e290364f39ac4ed90570d5aba7 P 85daee2113b445e293681247247f7f96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:15:55.024786 13602 raft_consensus.cc:3037] T 23e203e290364f39ac4ed90570d5aba7 P 85daee2113b445e293681247247f7f96 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:15:55.025357 13609 raft_consensus.cc:504] T 78c07f56e3dc4737bde64c7c9d8d55a2 P 79c63483422d4228a8f1b74b871f426f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c63483422d4228a8f1b74b871f426f" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 44935 } }
I20240501 14:15:55.025488 13609 leader_election.cc:302] T 78c07f56e3dc4737bde64c7c9d8d55a2 P 79c63483422d4228a8f1b74b871f426f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79c63483422d4228a8f1b74b871f426f; no voters:
I20240501 14:15:55.025560 13602 raft_consensus.cc:504] T 23e203e290364f39ac4ed90570d5aba7 P 85daee2113b445e293681247247f7f96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85daee2113b445e293681247247f7f96" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 34901 } }
I20240501 14:15:55.025601 13609 leader_election.cc:288] T 78c07f56e3dc4737bde64c7c9d8d55a2 P 79c63483422d4228a8f1b74b871f426f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:15:55.025646 13611 raft_consensus.cc:2781] T 78c07f56e3dc4737bde64c7c9d8d55a2 P 79c63483422d4228a8f1b74b871f426f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:15:55.025676 13602 leader_election.cc:302] T 23e203e290364f39ac4ed90570d5aba7 P 85daee2113b445e293681247247f7f96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85daee2113b445e293681247247f7f96; no voters:
I20240501 14:15:55.025748 13611 raft_consensus.cc:686] T 78c07f56e3dc4737bde64c7c9d8d55a2 P 79c63483422d4228a8f1b74b871f426f [term 1 LEADER]: Becoming Leader. State: Replica: 79c63483422d4228a8f1b74b871f426f, State: Running, Role: LEADER
I20240501 14:15:55.025854 13611 consensus_queue.cc:237] T 78c07f56e3dc4737bde64c7c9d8d55a2 P 79c63483422d4228a8f1b74b871f426f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c63483422d4228a8f1b74b871f426f" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 44935 } }
I20240501 14:15:55.025899 13609 ts_tablet_manager.cc:1398] T 78c07f56e3dc4737bde64c7c9d8d55a2 P 79c63483422d4228a8f1b74b871f426f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:15:55.026027 13609 tablet_bootstrap.cc:492] T edc2eeb4e0864195b6a709a525a48c54 P 79c63483422d4228a8f1b74b871f426f: Bootstrap starting.
I20240501 14:15:55.026013 13606 raft_consensus.cc:2781] T 23e203e290364f39ac4ed90570d5aba7 P 85daee2113b445e293681247247f7f96 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:15:55.026100 13606 raft_consensus.cc:686] T 23e203e290364f39ac4ed90570d5aba7 P 85daee2113b445e293681247247f7f96 [term 1 LEADER]: Becoming Leader. State: Replica: 85daee2113b445e293681247247f7f96, State: Running, Role: LEADER
I20240501 14:15:55.026202 13606 consensus_queue.cc:237] T 23e203e290364f39ac4ed90570d5aba7 P 85daee2113b445e293681247247f7f96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85daee2113b445e293681247247f7f96" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 34901 } }
I20240501 14:15:55.026873 13609 tablet_bootstrap.cc:654] T edc2eeb4e0864195b6a709a525a48c54 P 79c63483422d4228a8f1b74b871f426f: Neither blocks nor log segments found. Creating new log.
I20240501 14:15:55.027096 13602 leader_election.cc:288] T 23e203e290364f39ac4ed90570d5aba7 P 85daee2113b445e293681247247f7f96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:15:55.027417 13602 ts_tablet_manager.cc:1398] T 23e203e290364f39ac4ed90570d5aba7 P 85daee2113b445e293681247247f7f96: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:15:55.027657 12991 catalog_manager.cc:5484] T 78c07f56e3dc4737bde64c7c9d8d55a2 P 79c63483422d4228a8f1b74b871f426f reported cstate change: term changed from 0 to 1, leader changed from <none> to 79c63483422d4228a8f1b74b871f426f (127.7.39.130). New cstate: current_term: 1 leader_uuid: "79c63483422d4228a8f1b74b871f426f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c63483422d4228a8f1b74b871f426f" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 44935 } } }
I20240501 14:15:55.027935 13609 tablet_bootstrap.cc:492] T edc2eeb4e0864195b6a709a525a48c54 P 79c63483422d4228a8f1b74b871f426f: No bootstrap required, opened a new log
I20240501 14:15:55.027992 13609 ts_tablet_manager.cc:1367] T edc2eeb4e0864195b6a709a525a48c54 P 79c63483422d4228a8f1b74b871f426f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:15:55.027668 12990 catalog_manager.cc:5484] T 23e203e290364f39ac4ed90570d5aba7 P 85daee2113b445e293681247247f7f96 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85daee2113b445e293681247247f7f96 (127.7.39.129). New cstate: current_term: 1 leader_uuid: "85daee2113b445e293681247247f7f96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85daee2113b445e293681247247f7f96" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 34901 } } }
I20240501 14:15:55.028383 13609 raft_consensus.cc:348] T edc2eeb4e0864195b6a709a525a48c54 P 79c63483422d4228a8f1b74b871f426f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c63483422d4228a8f1b74b871f426f" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 44935 } }
I20240501 14:15:55.028443 13609 raft_consensus.cc:374] T edc2eeb4e0864195b6a709a525a48c54 P 79c63483422d4228a8f1b74b871f426f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:15:55.028470 13609 raft_consensus.cc:729] T edc2eeb4e0864195b6a709a525a48c54 P 79c63483422d4228a8f1b74b871f426f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79c63483422d4228a8f1b74b871f426f, State: Initialized, Role: FOLLOWER
I20240501 14:15:55.028559 13609 consensus_queue.cc:260] T edc2eeb4e0864195b6a709a525a48c54 P 79c63483422d4228a8f1b74b871f426f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c63483422d4228a8f1b74b871f426f" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 44935 } }
I20240501 14:15:55.028630 13609 raft_consensus.cc:388] T edc2eeb4e0864195b6a709a525a48c54 P 79c63483422d4228a8f1b74b871f426f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:15:55.028656 13609 raft_consensus.cc:482] T edc2eeb4e0864195b6a709a525a48c54 P 79c63483422d4228a8f1b74b871f426f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:15:55.028692 13609 raft_consensus.cc:3037] T edc2eeb4e0864195b6a709a525a48c54 P 79c63483422d4228a8f1b74b871f426f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:15:55.029395 13609 raft_consensus.cc:504] T edc2eeb4e0864195b6a709a525a48c54 P 79c63483422d4228a8f1b74b871f426f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c63483422d4228a8f1b74b871f426f" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 44935 } }
I20240501 14:15:55.029497 13609 leader_election.cc:302] T edc2eeb4e0864195b6a709a525a48c54 P 79c63483422d4228a8f1b74b871f426f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79c63483422d4228a8f1b74b871f426f; no voters:
I20240501 14:15:55.029563 13609 leader_election.cc:288] T edc2eeb4e0864195b6a709a525a48c54 P 79c63483422d4228a8f1b74b871f426f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:15:55.029744 13609 ts_tablet_manager.cc:1398] T edc2eeb4e0864195b6a709a525a48c54 P 79c63483422d4228a8f1b74b871f426f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:15:55.029783 13612 raft_consensus.cc:2781] T edc2eeb4e0864195b6a709a525a48c54 P 79c63483422d4228a8f1b74b871f426f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:15:55.029845 13612 raft_consensus.cc:686] T edc2eeb4e0864195b6a709a525a48c54 P 79c63483422d4228a8f1b74b871f426f [term 1 LEADER]: Becoming Leader. State: Replica: 79c63483422d4228a8f1b74b871f426f, State: Running, Role: LEADER
I20240501 14:15:55.029960 13612 consensus_queue.cc:237] T edc2eeb4e0864195b6a709a525a48c54 P 79c63483422d4228a8f1b74b871f426f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c63483422d4228a8f1b74b871f426f" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 44935 } }
I20240501 14:15:55.032102 12991 catalog_manager.cc:5484] T edc2eeb4e0864195b6a709a525a48c54 P 79c63483422d4228a8f1b74b871f426f reported cstate change: term changed from 0 to 1, leader changed from <none> to 79c63483422d4228a8f1b74b871f426f (127.7.39.130). New cstate: current_term: 1 leader_uuid: "79c63483422d4228a8f1b74b871f426f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c63483422d4228a8f1b74b871f426f" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 44935 } } }
I20240501 14:15:55.181746 12991 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50778:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:15:55.186630 13113 tablet_service.cc:1467] Processing CreateTablet for tablet 3c02aa4af139477a8cabffd891dda25c (DEFAULT_TABLE table=rebalance_test_table_3 [id=f782e2146b5544eb9c5b316520275979]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:15:55.186666 13243 tablet_service.cc:1467] Processing CreateTablet for tablet 059255f52feb456da77c61dec9546aa1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=f782e2146b5544eb9c5b316520275979]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:15:55.186936 13113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c02aa4af139477a8cabffd891dda25c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:15:55.186976 13243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 059255f52feb456da77c61dec9546aa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:15:55.188627 13609 tablet_bootstrap.cc:492] T 059255f52feb456da77c61dec9546aa1 P 79c63483422d4228a8f1b74b871f426f: Bootstrap starting.
I20240501 14:15:55.188997 13113 tablet_service.cc:1467] Processing CreateTablet for tablet e843e6bf06694116bb62fbf9edc75a7a (DEFAULT_TABLE table=rebalance_test_table_3 [id=f782e2146b5544eb9c5b316520275979]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:15:55.189232 13113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e843e6bf06694116bb62fbf9edc75a7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:15:55.189576 13609 tablet_bootstrap.cc:654] T 059255f52feb456da77c61dec9546aa1 P 79c63483422d4228a8f1b74b871f426f: Neither blocks nor log segments found. Creating new log.
I20240501 14:15:55.190428 13602 tablet_bootstrap.cc:492] T 3c02aa4af139477a8cabffd891dda25c P 85daee2113b445e293681247247f7f96: Bootstrap starting.
I20240501 14:15:55.190934 13609 tablet_bootstrap.cc:492] T 059255f52feb456da77c61dec9546aa1 P 79c63483422d4228a8f1b74b871f426f: No bootstrap required, opened a new log
I20240501 14:15:55.191020 13609 ts_tablet_manager.cc:1367] T 059255f52feb456da77c61dec9546aa1 P 79c63483422d4228a8f1b74b871f426f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:15:55.191401 13602 tablet_bootstrap.cc:654] T 3c02aa4af139477a8cabffd891dda25c P 85daee2113b445e293681247247f7f96: Neither blocks nor log segments found. Creating new log.
I20240501 14:15:55.191445 13609 raft_consensus.cc:348] T 059255f52feb456da77c61dec9546aa1 P 79c63483422d4228a8f1b74b871f426f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c63483422d4228a8f1b74b871f426f" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 44935 } }
I20240501 14:15:55.191547 13609 raft_consensus.cc:374] T 059255f52feb456da77c61dec9546aa1 P 79c63483422d4228a8f1b74b871f426f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:15:55.191592 13609 raft_consensus.cc:729] T 059255f52feb456da77c61dec9546aa1 P 79c63483422d4228a8f1b74b871f426f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79c63483422d4228a8f1b74b871f426f, State: Initialized, Role: FOLLOWER
I20240501 14:15:55.191717 13609 consensus_queue.cc:260] T 059255f52feb456da77c61dec9546aa1 P 79c63483422d4228a8f1b74b871f426f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c63483422d4228a8f1b74b871f426f" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 44935 } }
I20240501 14:15:55.191790 13609 raft_consensus.cc:388] T 059255f52feb456da77c61dec9546aa1 P 79c63483422d4228a8f1b74b871f426f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:15:55.191814 13609 raft_consensus.cc:482] T 059255f52feb456da77c61dec9546aa1 P 79c63483422d4228a8f1b74b871f426f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:15:55.191866 13609 raft_consensus.cc:3037] T 059255f52feb456da77c61dec9546aa1 P 79c63483422d4228a8f1b74b871f426f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:15:55.192457 13602 tablet_bootstrap.cc:492] T 3c02aa4af139477a8cabffd891dda25c P 85daee2113b445e293681247247f7f96: No bootstrap required, opened a new log
I20240501 14:15:55.192543 13602 ts_tablet_manager.cc:1367] T 3c02aa4af139477a8cabffd891dda25c P 85daee2113b445e293681247247f7f96: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:15:55.192715 13609 raft_consensus.cc:504] T 059255f52feb456da77c61dec9546aa1 P 79c63483422d4228a8f1b74b871f426f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c63483422d4228a8f1b74b871f426f" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 44935 } }
I20240501 14:15:55.192840 13609 leader_election.cc:302] T 059255f52feb456da77c61dec9546aa1 P 79c63483422d4228a8f1b74b871f426f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79c63483422d4228a8f1b74b871f426f; no voters:
I20240501 14:15:55.192947 13609 leader_election.cc:288] T 059255f52feb456da77c61dec9546aa1 P 79c63483422d4228a8f1b74b871f426f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:15:55.193014 13613 raft_consensus.cc:2781] T 059255f52feb456da77c61dec9546aa1 P 79c63483422d4228a8f1b74b871f426f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:15:55.193130 13613 raft_consensus.cc:686] T 059255f52feb456da77c61dec9546aa1 P 79c63483422d4228a8f1b74b871f426f [term 1 LEADER]: Becoming Leader. State: Replica: 79c63483422d4228a8f1b74b871f426f, State: Running, Role: LEADER
I20240501 14:15:55.193104 13602 raft_consensus.cc:348] T 3c02aa4af139477a8cabffd891dda25c P 85daee2113b445e293681247247f7f96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85daee2113b445e293681247247f7f96" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 34901 } }
I20240501 14:15:55.193184 13609 ts_tablet_manager.cc:1398] T 059255f52feb456da77c61dec9546aa1 P 79c63483422d4228a8f1b74b871f426f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:15:55.193204 13602 raft_consensus.cc:374] T 3c02aa4af139477a8cabffd891dda25c P 85daee2113b445e293681247247f7f96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:15:55.193250 13602 raft_consensus.cc:729] T 3c02aa4af139477a8cabffd891dda25c P 85daee2113b445e293681247247f7f96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85daee2113b445e293681247247f7f96, State: Initialized, Role: FOLLOWER
I20240501 14:15:55.193277 13613 consensus_queue.cc:237] T 059255f52feb456da77c61dec9546aa1 P 79c63483422d4228a8f1b74b871f426f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c63483422d4228a8f1b74b871f426f" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 44935 } }
I20240501 14:15:55.193358 13602 consensus_queue.cc:260] T 3c02aa4af139477a8cabffd891dda25c P 85daee2113b445e293681247247f7f96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85daee2113b445e293681247247f7f96" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 34901 } }
I20240501 14:15:55.193436 13602 raft_consensus.cc:388] T 3c02aa4af139477a8cabffd891dda25c P 85daee2113b445e293681247247f7f96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:15:55.193475 13602 raft_consensus.cc:482] T 3c02aa4af139477a8cabffd891dda25c P 85daee2113b445e293681247247f7f96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:15:55.193521 13602 raft_consensus.cc:3037] T 3c02aa4af139477a8cabffd891dda25c P 85daee2113b445e293681247247f7f96 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:15:55.194284 13602 raft_consensus.cc:504] T 3c02aa4af139477a8cabffd891dda25c P 85daee2113b445e293681247247f7f96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85daee2113b445e293681247247f7f96" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 34901 } }
I20240501 14:15:55.194402 13602 leader_election.cc:302] T 3c02aa4af139477a8cabffd891dda25c P 85daee2113b445e293681247247f7f96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85daee2113b445e293681247247f7f96; no voters:
I20240501 14:15:55.194553 13602 leader_election.cc:288] T 3c02aa4af139477a8cabffd891dda25c P 85daee2113b445e293681247247f7f96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:15:55.194599 13604 raft_consensus.cc:2781] T 3c02aa4af139477a8cabffd891dda25c P 85daee2113b445e293681247247f7f96 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:15:55.194546 12991 catalog_manager.cc:5484] T 059255f52feb456da77c61dec9546aa1 P 79c63483422d4228a8f1b74b871f426f reported cstate change: term changed from 0 to 1, leader changed from <none> to 79c63483422d4228a8f1b74b871f426f (127.7.39.130). New cstate: current_term: 1 leader_uuid: "79c63483422d4228a8f1b74b871f426f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c63483422d4228a8f1b74b871f426f" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 44935 } } }
I20240501 14:15:55.194684 13604 raft_consensus.cc:686] T 3c02aa4af139477a8cabffd891dda25c P 85daee2113b445e293681247247f7f96 [term 1 LEADER]: Becoming Leader. State: Replica: 85daee2113b445e293681247247f7f96, State: Running, Role: LEADER
I20240501 14:15:55.194802 13604 consensus_queue.cc:237] T 3c02aa4af139477a8cabffd891dda25c P 85daee2113b445e293681247247f7f96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85daee2113b445e293681247247f7f96" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 34901 } }
I20240501 14:15:55.194861 13602 ts_tablet_manager.cc:1398] T 3c02aa4af139477a8cabffd891dda25c P 85daee2113b445e293681247247f7f96: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:15:55.195021 13602 tablet_bootstrap.cc:492] T e843e6bf06694116bb62fbf9edc75a7a P 85daee2113b445e293681247247f7f96: Bootstrap starting.
I20240501 14:15:55.195953 13602 tablet_bootstrap.cc:654] T e843e6bf06694116bb62fbf9edc75a7a P 85daee2113b445e293681247247f7f96: Neither blocks nor log segments found. Creating new log.
I20240501 14:15:55.196198 12989 catalog_manager.cc:5484] T 3c02aa4af139477a8cabffd891dda25c P 85daee2113b445e293681247247f7f96 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85daee2113b445e293681247247f7f96 (127.7.39.129). New cstate: current_term: 1 leader_uuid: "85daee2113b445e293681247247f7f96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85daee2113b445e293681247247f7f96" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 34901 } } }
I20240501 14:15:55.196910 13602 tablet_bootstrap.cc:492] T e843e6bf06694116bb62fbf9edc75a7a P 85daee2113b445e293681247247f7f96: No bootstrap required, opened a new log
I20240501 14:15:55.196974 13602 ts_tablet_manager.cc:1367] T e843e6bf06694116bb62fbf9edc75a7a P 85daee2113b445e293681247247f7f96: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:15:55.197367 13602 raft_consensus.cc:348] T e843e6bf06694116bb62fbf9edc75a7a P 85daee2113b445e293681247247f7f96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85daee2113b445e293681247247f7f96" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 34901 } }
I20240501 14:15:55.197458 13602 raft_consensus.cc:374] T e843e6bf06694116bb62fbf9edc75a7a P 85daee2113b445e293681247247f7f96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:15:55.197497 13602 raft_consensus.cc:729] T e843e6bf06694116bb62fbf9edc75a7a P 85daee2113b445e293681247247f7f96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85daee2113b445e293681247247f7f96, State: Initialized, Role: FOLLOWER
I20240501 14:15:55.197611 13602 consensus_queue.cc:260] T e843e6bf06694116bb62fbf9edc75a7a P 85daee2113b445e293681247247f7f96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85daee2113b445e293681247247f7f96" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 34901 } }
I20240501 14:15:55.197690 13602 raft_consensus.cc:388] T e843e6bf06694116bb62fbf9edc75a7a P 85daee2113b445e293681247247f7f96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:15:55.197719 13602 raft_consensus.cc:482] T e843e6bf06694116bb62fbf9edc75a7a P 85daee2113b445e293681247247f7f96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:15:55.197747 13602 raft_consensus.cc:3037] T e843e6bf06694116bb62fbf9edc75a7a P 85daee2113b445e293681247247f7f96 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:15:55.198495 13602 raft_consensus.cc:504] T e843e6bf06694116bb62fbf9edc75a7a P 85daee2113b445e293681247247f7f96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85daee2113b445e293681247247f7f96" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 34901 } }
I20240501 14:15:55.198601 13602 leader_election.cc:302] T e843e6bf06694116bb62fbf9edc75a7a P 85daee2113b445e293681247247f7f96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85daee2113b445e293681247247f7f96; no voters:
I20240501 14:15:55.198726 13602 leader_election.cc:288] T e843e6bf06694116bb62fbf9edc75a7a P 85daee2113b445e293681247247f7f96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:15:55.198793 13605 raft_consensus.cc:2781] T e843e6bf06694116bb62fbf9edc75a7a P 85daee2113b445e293681247247f7f96 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:15:55.198902 13605 raft_consensus.cc:686] T e843e6bf06694116bb62fbf9edc75a7a P 85daee2113b445e293681247247f7f96 [term 1 LEADER]: Becoming Leader. State: Replica: 85daee2113b445e293681247247f7f96, State: Running, Role: LEADER
I20240501 14:15:55.199009 13605 consensus_queue.cc:237] T e843e6bf06694116bb62fbf9edc75a7a P 85daee2113b445e293681247247f7f96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85daee2113b445e293681247247f7f96" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 34901 } }
I20240501 14:15:55.199095 13602 ts_tablet_manager.cc:1398] T e843e6bf06694116bb62fbf9edc75a7a P 85daee2113b445e293681247247f7f96: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:15:55.200388 12991 catalog_manager.cc:5484] T e843e6bf06694116bb62fbf9edc75a7a P 85daee2113b445e293681247247f7f96 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85daee2113b445e293681247247f7f96 (127.7.39.129). New cstate: current_term: 1 leader_uuid: "85daee2113b445e293681247247f7f96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85daee2113b445e293681247247f7f96" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 34901 } } }
I20240501 14:15:55.355158 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.131:33077
--local_ip_for_outbound_sockets=127.7.39.131
--tserver_master_addrs=127.7.39.190:37333
--webserver_port=34521
--webserver_interface=127.7.39.131
--builtin_ntp_servers=127.7.39.148:33663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 14:15:55.452260 13708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:15:55.452452 13708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:15:55.452492 13708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:15:55.455149 13708 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:15:55.455196 13708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:15:55.455251 13708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.131
I20240501 14:15:55.458475 13708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:33663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.131:33077
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.7.39.131
--webserver_port=34521
--tserver_master_addrs=127.7.39.190:37333
--never_fsync=true
--heap_profile_path=/tmp/kudu.13708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.131
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:15:55.459136 13708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:15:55.460031 13708 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:15:55.467420 13713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:15:55.468081 13714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:15:55.468601 13716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:15:55.469244 13708 server_base.cc:1034] running on GCE node
I20240501 14:15:55.469578 13708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:15:55.470067 13708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:15:55.471218 13708 hybrid_clock.cc:648] HybridClock initialized: now 1714572955471189 us; error 34 us; skew 500 ppm
I20240501 14:15:55.472885 13708 webserver.cc:460] Webserver started at http://127.7.39.131:34521/ using document root <none> and password file <none>
I20240501 14:15:55.473369 13708 fs_manager.cc:359] Metadata directory not provided
I20240501 14:15:55.473440 13708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:15:55.476450 13708 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:15:55.478051 13722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:15:55.478889 13708 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:15:55.478983 13708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "76cb2fb202294a2d87080b62ca189bb9"
format_stamp: "Formatted at 2024-05-01 14:15:50 on dist-test-slave-rnfw"
I20240501 14:15:55.479337 13708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:15:55.503202 13708 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:15:55.503688 13708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:15:55.503837 13708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:15:55.504315 13708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:15:55.505072 13708 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:15:55.505120 13708 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:15:55.505172 13708 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:15:55.505198 13708 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:15:55.512848 13708 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.131:33077
I20240501 14:15:55.512964 13835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.131:33077 every 8 connection(s)
I20240501 14:15:55.513728 13708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 14:15:55.520538 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 13708
I20240501 14:15:55.521044 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.132:39407
--local_ip_for_outbound_sockets=127.7.39.132
--tserver_master_addrs=127.7.39.190:37333
--webserver_port=33535
--webserver_interface=127.7.39.132
--builtin_ntp_servers=127.7.39.148:33663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:15:55.521461 13836 heartbeater.cc:347] Connected to a master server at 127.7.39.190:37333
I20240501 14:15:55.521642 13836 heartbeater.cc:464] Registering TS with master...
I20240501 14:15:55.522001 13836 heartbeater.cc:511] Master 127.7.39.190:37333 requested a full tablet report, sending...
I20240501 14:15:55.522933 12991 ts_manager.cc:194] Re-registered known tserver with Master: 76cb2fb202294a2d87080b62ca189bb9 (127.7.39.131:33077)
I20240501 14:15:55.523481 12991 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.131:40713
W20240501 14:15:55.621421 13839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:15:55.621618 13839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:15:55.621656 13839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:15:55.624568 13839 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:15:55.624627 13839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:15:55.624708 13839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.132
I20240501 14:15:55.627974 13839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:33663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.132:39407
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.7.39.132
--webserver_port=33535
--tserver_master_addrs=127.7.39.190:37333
--never_fsync=true
--heap_profile_path=/tmp/kudu.13839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.132
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:15:55.628695 13839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:15:55.629647 13839 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:15:55.637544 13844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:15:55.638159 13847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:15:55.638787 13845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:15:55.639429 13839 server_base.cc:1034] running on GCE node
I20240501 14:15:55.639786 13839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:15:55.640369 13839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:15:55.641526 13839 hybrid_clock.cc:648] HybridClock initialized: now 1714572955641513 us; error 38 us; skew 500 ppm
I20240501 14:15:55.643270 13839 webserver.cc:460] Webserver started at http://127.7.39.132:33535/ using document root <none> and password file <none>
I20240501 14:15:55.643750 13839 fs_manager.cc:359] Metadata directory not provided
I20240501 14:15:55.643824 13839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:15:55.646816 13839 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:15:55.648516 13853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:15:55.649422 13839 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 14:15:55.649523 13839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "9b9e9b6a68404d638f1edeeefe7c2633"
format_stamp: "Formatted at 2024-05-01 14:15:50 on dist-test-slave-rnfw"
I20240501 14:15:55.649885 13839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:15:55.675853 13839 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:15:55.676326 13839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:15:55.676461 13839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:15:55.676923 13839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:15:55.677685 13839 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:15:55.677729 13839 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:15:55.677791 13839 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:15:55.677817 13839 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:15:55.686215 13839 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.132:39407
I20240501 14:15:55.686260 13966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.132:39407 every 8 connection(s)
I20240501 14:15:55.687196 13839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 14:15:55.695108 13967 heartbeater.cc:347] Connected to a master server at 127.7.39.190:37333
I20240501 14:15:55.695293 13967 heartbeater.cc:464] Registering TS with master...
I20240501 14:15:55.695650 13967 heartbeater.cc:511] Master 127.7.39.190:37333 requested a full tablet report, sending...
I20240501 14:15:55.696468 12991 ts_manager.cc:194] Re-registered known tserver with Master: 9b9e9b6a68404d638f1edeeefe7c2633 (127.7.39.132:39407)
I20240501 14:15:55.697022 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 13839
I20240501 14:15:55.697080 12991 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.132:57769
I20240501 14:15:55.871631 13113 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:15:55.874142 13243 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:55.875577 13770 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:15:55.885881 13901 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:15:55.963289 13770 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:15:55.963475 13901 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:15:55.963701 13243 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:55.964397 13113 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:15:56.063685 13901 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:15:56.069538 13770 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:15:56.069576 13113 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:15:56.088680 13243 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:56.162325 13770 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:15:56.163034 13113 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:15:56.165670 13243 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:15:56.171216 13901 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
4e0fceaf25e84af7832884004d8d5cb7 | 127.7.39.190:37333 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.39.148:33663 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
76cb2fb202294a2d87080b62ca189bb9 | 127.7.39.131:33077 | HEALTHY | <none> | 0 | 0
79c63483422d4228a8f1b74b871f426f | 127.7.39.130:44935 | HEALTHY | <none> | 6 | 0
85daee2113b445e293681247247f7f96 | 127.7.39.129:34901 | HEALTHY | <none> | 7 | 0
9b9e9b6a68404d638f1edeeefe7c2633 | 127.7.39.132:39407 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 4 server(s) checked
local_ip_for_outbound_sockets | 127.7.39.129 | experimental | 127.7.39.129:34901
local_ip_for_outbound_sockets | 127.7.39.130 | experimental | 127.7.39.130:44935
local_ip_for_outbound_sockets | 127.7.39.131 | experimental | 127.7.39.131:33077
local_ip_for_outbound_sockets | 127.7.39.132 | experimental | 127.7.39.132:39407
never_fsync | true | unsafe,advanced | all 4 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 4 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 4 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
server_dump_info_format | pb | hidden | all 4 server(s) checked
server_dump_info_path | /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.7.39.129:34901
server_dump_info_path | /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.39.130:44935
server_dump_info_path | /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.39.131:33077
server_dump_info_path | /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.7.39.132:39407
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.39.148:33663 | all 4 server(s) checked
time_source | builtin | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 6
Third Quartile | 7
Maximum | 7
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 5
Tablets | 13
Replicas | 13
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 14:15:56.189699 7326 log_verifier.cc:126] Checking tablet 022b6a678a1b46a8b8fe6aae03cda95f
I20240501 14:15:56.197443 7326 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 022b6a678a1b46a8b8fe6aae03cda95f
I20240501 14:15:56.197494 7326 log_verifier.cc:126] Checking tablet 059255f52feb456da77c61dec9546aa1
I20240501 14:15:56.205966 7326 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 059255f52feb456da77c61dec9546aa1
I20240501 14:15:56.206041 7326 log_verifier.cc:126] Checking tablet 23e203e290364f39ac4ed90570d5aba7
I20240501 14:15:56.213373 7326 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 23e203e290364f39ac4ed90570d5aba7
I20240501 14:15:56.213435 7326 log_verifier.cc:126] Checking tablet 3a40e3962fa442d49b12592b9761a73d
I20240501 14:15:56.220679 7326 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 3a40e3962fa442d49b12592b9761a73d
I20240501 14:15:56.220736 7326 log_verifier.cc:126] Checking tablet 3c02aa4af139477a8cabffd891dda25c
I20240501 14:15:56.227973 7326 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 3c02aa4af139477a8cabffd891dda25c
I20240501 14:15:56.228021 7326 log_verifier.cc:126] Checking tablet 78c07f56e3dc4737bde64c7c9d8d55a2
I20240501 14:15:56.235637 7326 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 78c07f56e3dc4737bde64c7c9d8d55a2
I20240501 14:15:56.235687 7326 log_verifier.cc:126] Checking tablet a94adcc02f9b4de8a20bb59c4d291485
I20240501 14:15:56.242879 7326 log_verifier.cc:177] Verified matching terms for 2 ops in tablet a94adcc02f9b4de8a20bb59c4d291485
I20240501 14:15:56.242928 7326 log_verifier.cc:126] Checking tablet a9807ddd18aa48e69bcfcc7def51d1b0
I20240501 14:15:56.250198 7326 log_verifier.cc:177] Verified matching terms for 2 ops in tablet a9807ddd18aa48e69bcfcc7def51d1b0
I20240501 14:15:56.250250 7326 log_verifier.cc:126] Checking tablet b02f666697f34d15a1ae1a38c0e44cd6
I20240501 14:15:56.257670 7326 log_verifier.cc:177] Verified matching terms for 1 ops in tablet b02f666697f34d15a1ae1a38c0e44cd6
I20240501 14:15:56.257722 7326 log_verifier.cc:126] Checking tablet cf57e7aaa70a43afb0bdfd0eb37f0709
I20240501 14:15:56.265064 7326 log_verifier.cc:177] Verified matching terms for 1 ops in tablet cf57e7aaa70a43afb0bdfd0eb37f0709
I20240501 14:15:56.265115 7326 log_verifier.cc:126] Checking tablet e843e6bf06694116bb62fbf9edc75a7a
I20240501 14:15:56.272327 7326 log_verifier.cc:177] Verified matching terms for 1 ops in tablet e843e6bf06694116bb62fbf9edc75a7a
I20240501 14:15:56.272377 7326 log_verifier.cc:126] Checking tablet edc2eeb4e0864195b6a709a525a48c54
I20240501 14:15:56.279537 7326 log_verifier.cc:177] Verified matching terms for 1 ops in tablet edc2eeb4e0864195b6a709a525a48c54
I20240501 14:15:56.279587 7326 log_verifier.cc:126] Checking tablet fac046d9307f4f6ab55169083f3c0211
I20240501 14:15:56.286892 7326 log_verifier.cc:177] Verified matching terms for 1 ops in tablet fac046d9307f4f6ab55169083f3c0211
I20240501 14:15:56.287359 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 13051
I20240501 14:15:56.294283 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 13182
I20240501 14:15:56.300429 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 13708
I20240501 14:15:56.305792 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 13839
I20240501 14:15:56.311272 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 12960
2024-05-01T14:15:56Z chronyd exiting
[ OK ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/0 (6451 ms)
[----------] 1 test from RebalancerAndSingleReplicaTablets (6451 ms total)
[----------] 2 tests from LocationAwareRebalancingParamTest
[ RUN ] LocationAwareRebalancingParamTest.Rebalance/idx2_rf3_t3_l_3_2
I20240501 14:15:56.345286 7326 test_util.cc:268] Using random seed: 536945602
I20240501 14:15:56.346006 7326 ts_itest-base.cc:112] Starting cluster with:
I20240501 14:15:56.346041 7326 ts_itest-base.cc:113] --------------
I20240501 14:15:56.346053 7326 ts_itest-base.cc:114] 5 tablet servers
I20240501 14:15:56.346063 7326 ts_itest-base.cc:115] 3 replicas per TS
I20240501 14:15:56.346073 7326 ts_itest-base.cc:116] --------------
2024-05-01T14:15:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T14:15:56Z Disabled control of system clock
I20240501 14:15:56.355937 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.39.190:36791
--webserver_interface=127.7.39.190
--webserver_port=0
--builtin_ntp_servers=127.7.39.148:34331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.39.190:36791
--location_mapping_cmd=/tmp/dist-test-taskhUqsdZ/build/debug/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:2
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240501 14:15:56.455864 14113 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240501 14:15:56.456079 14113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:15:56.456108 14113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:15:56.456166 14113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:15:56.459002 14113 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:15:56.459096 14113 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 14:15:56.459143 14113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:15:56.459177 14113 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 14:15:56.459213 14113 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 14:15:56.462630 14113 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:34331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskhUqsdZ/build/debug/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:2
--ipki_ca_key_size=768
--master_addresses=127.7.39.190:36791
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.39.190:36791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.7.39.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.14113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:15:56.463333 14113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:15:56.464255 14113 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:15:56.470746 14118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:15:56.471226 14119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:15:56.471904 14121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:15:56.472532 14113 server_base.cc:1034] running on GCE node
I20240501 14:15:56.472924 14113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:15:56.473721 14113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:15:56.474866 14113 hybrid_clock.cc:648] HybridClock initialized: now 1714572956474859 us; error 23 us; skew 500 ppm
I20240501 14:15:56.476423 14113 webserver.cc:460] Webserver started at http://127.7.39.190:38491/ using document root <none> and password file <none>
I20240501 14:15:56.476898 14113 fs_manager.cc:359] Metadata directory not provided
I20240501 14:15:56.476953 14113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:15:56.477130 14113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:15:56.478576 14113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "4a4a60356ff44ef5933cad67de7da2bd"
format_stamp: "Formatted at 2024-05-01 14:15:56 on dist-test-slave-rnfw"
I20240501 14:15:56.479019 14113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "4a4a60356ff44ef5933cad67de7da2bd"
format_stamp: "Formatted at 2024-05-01 14:15:56 on dist-test-slave-rnfw"
I20240501 14:15:56.482002 14113 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240501 14:15:56.483906 14127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:15:56.484732 14113 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 14:15:56.484846 14113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/wal
uuid: "4a4a60356ff44ef5933cad67de7da2bd"
format_stamp: "Formatted at 2024-05-01 14:15:56 on dist-test-slave-rnfw"
I20240501 14:15:56.484917 14113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:15:56.508502 14113 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:15:56.509058 14113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:15:56.509199 14113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:15:56.515354 14113 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.190:36791
I20240501 14:15:56.515369 14179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.190:36791 every 8 connection(s)
I20240501 14:15:56.516184 14113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 14:15:56.518437 14180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:15:56.520462 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 14113
I20240501 14:15:56.520591 7326 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 14:15:56.523350 14180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a4a60356ff44ef5933cad67de7da2bd: Bootstrap starting.
I20240501 14:15:56.525141 14180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a4a60356ff44ef5933cad67de7da2bd: Neither blocks nor log segments found. Creating new log.
I20240501 14:15:56.525758 14180 log.cc:824] T 00000000000000000000000000000000 P 4a4a60356ff44ef5933cad67de7da2bd: Log is configured to *not* fsync() on all Append() calls
I20240501 14:15:56.527294 14180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a4a60356ff44ef5933cad67de7da2bd: No bootstrap required, opened a new log
I20240501 14:15:56.529644 14180 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a4a60356ff44ef5933cad67de7da2bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a4a60356ff44ef5933cad67de7da2bd" member_type: VOTER last_known_addr { host: "127.7.39.190" port: 36791 } }
I20240501 14:15:56.529796 14180 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a4a60356ff44ef5933cad67de7da2bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:15:56.529841 14180 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a4a60356ff44ef5933cad67de7da2bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a4a60356ff44ef5933cad67de7da2bd, State: Initialized, Role: FOLLOWER
I20240501 14:15:56.530161 14180 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a4a60356ff44ef5933cad67de7da2bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a4a60356ff44ef5933cad67de7da2bd" member_type: VOTER last_known_addr { host: "127.7.39.190" port: 36791 } }
I20240501 14:15:56.530267 14180 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a4a60356ff44ef5933cad67de7da2bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:15:56.530309 14180 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a4a60356ff44ef5933cad67de7da2bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:15:56.530381 14180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a4a60356ff44ef5933cad67de7da2bd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:15:56.530975 14180 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a4a60356ff44ef5933cad67de7da2bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a4a60356ff44ef5933cad67de7da2bd" member_type: VOTER last_known_addr { host: "127.7.39.190" port: 36791 } }
I20240501 14:15:56.531287 14180 leader_election.cc:302] T 00000000000000000000000000000000 P 4a4a60356ff44ef5933cad67de7da2bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a4a60356ff44ef5933cad67de7da2bd; no voters:
I20240501 14:15:56.531517 14180 leader_election.cc:288] T 00000000000000000000000000000000 P 4a4a60356ff44ef5933cad67de7da2bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:15:56.531602 14185 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a4a60356ff44ef5933cad67de7da2bd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:15:56.532143 14185 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a4a60356ff44ef5933cad67de7da2bd [term 1 LEADER]: Becoming Leader. State: Replica: 4a4a60356ff44ef5933cad67de7da2bd, State: Running, Role: LEADER
I20240501 14:15:56.532411 14180 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a4a60356ff44ef5933cad67de7da2bd [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:15:56.532392 14185 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a4a60356ff44ef5933cad67de7da2bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a4a60356ff44ef5933cad67de7da2bd" member_type: VOTER last_known_addr { host: "127.7.39.190" port: 36791 } }
I20240501 14:15:56.533566 14187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a4a60356ff44ef5933cad67de7da2bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a4a60356ff44ef5933cad67de7da2bd. Latest consensus state: current_term: 1 leader_uuid: "4a4a60356ff44ef5933cad67de7da2bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a4a60356ff44ef5933cad67de7da2bd" member_type: VOTER last_known_addr { host: "127.7.39.190" port: 36791 } } }
I20240501 14:15:56.533675 14187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a4a60356ff44ef5933cad67de7da2bd [sys.catalog]: This master's current role is: LEADER
I20240501 14:15:56.534291 14195 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:15:56.534451 14186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a4a60356ff44ef5933cad67de7da2bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a4a60356ff44ef5933cad67de7da2bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a4a60356ff44ef5933cad67de7da2bd" member_type: VOTER last_known_addr { host: "127.7.39.190" port: 36791 } } }
I20240501 14:15:56.534705 14186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a4a60356ff44ef5933cad67de7da2bd [sys.catalog]: This master's current role is: LEADER
I20240501 14:15:56.536942 14195 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:15:56.541615 14195 catalog_manager.cc:1347] Generated new cluster ID: 058a2cc4f3f547d788c506a7f2fb035f
I20240501 14:15:56.541723 14195 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:15:56.560873 14195 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 14:15:56.561677 14195 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:15:56.570776 14195 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 4a4a60356ff44ef5933cad67de7da2bd: Generated new TSK 0
I20240501 14:15:56.571388 14195 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:15:56.579727 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.129:0
--local_ip_for_outbound_sockets=127.7.39.129
--webserver_interface=127.7.39.129
--webserver_port=0
--tserver_master_addrs=127.7.39.190:36791
--builtin_ntp_servers=127.7.39.148:34331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 14:15:56.681182 14204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:15:56.681385 14204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:15:56.681429 14204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:15:56.684170 14204 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:15:56.684239 14204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:15:56.684330 14204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.129
I20240501 14:15:56.687605 14204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:34331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.7.39.129
--webserver_port=0
--tserver_master_addrs=127.7.39.190:36791
--never_fsync=true
--heap_profile_path=/tmp/kudu.14204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.129
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:15:56.688275 14204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:15:56.689206 14204 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:15:56.696679 14209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:15:56.697255 14212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:15:56.697909 14210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:15:56.698601 14204 server_base.cc:1034] running on GCE node
I20240501 14:15:56.698886 14204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:15:56.699371 14204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:15:56.700501 14204 hybrid_clock.cc:648] HybridClock initialized: now 1714572956700495 us; error 24 us; skew 500 ppm
I20240501 14:15:56.701965 14204 webserver.cc:460] Webserver started at http://127.7.39.129:43101/ using document root <none> and password file <none>
I20240501 14:15:56.702461 14204 fs_manager.cc:359] Metadata directory not provided
I20240501 14:15:56.702514 14204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:15:56.702675 14204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:15:56.704126 14204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "c87cbbf2d3ab4143815ef2c74ae760db"
format_stamp: "Formatted at 2024-05-01 14:15:56 on dist-test-slave-rnfw"
I20240501 14:15:56.704557 14204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "c87cbbf2d3ab4143815ef2c74ae760db"
format_stamp: "Formatted at 2024-05-01 14:15:56 on dist-test-slave-rnfw"
I20240501 14:15:56.707327 14204 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:15:56.709122 14218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:15:56.709937 14204 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:15:56.710033 14204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "c87cbbf2d3ab4143815ef2c74ae760db"
format_stamp: "Formatted at 2024-05-01 14:15:56 on dist-test-slave-rnfw"
I20240501 14:15:56.710101 14204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:15:56.724474 14204 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:15:56.724890 14204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:15:56.725014 14204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:15:56.725419 14204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:15:56.726130 14204 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:15:56.726177 14204 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:15:56.726228 14204 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:15:56.726253 14204 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:15:56.734141 14204 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.129:42369
I20240501 14:15:56.734181 14331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.129:42369 every 8 connection(s)
I20240501 14:15:56.734876 14204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 14:15:56.735178 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 14204
I20240501 14:15:56.735316 7326 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 14:15:56.737438 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.130:0
--local_ip_for_outbound_sockets=127.7.39.130
--webserver_interface=127.7.39.130
--webserver_port=0
--tserver_master_addrs=127.7.39.190:36791
--builtin_ntp_servers=127.7.39.148:34331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:15:56.743536 14332 heartbeater.cc:347] Connected to a master server at 127.7.39.190:36791
I20240501 14:15:56.743737 14332 heartbeater.cc:464] Registering TS with master...
I20240501 14:15:56.744170 14332 heartbeater.cc:511] Master 127.7.39.190:36791 requested a full tablet report, sending...
I20240501 14:15:56.779937 14144 ts_manager.cc:194] Registered new tserver with Master: c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129:42369)
I20240501 14:15:56.781747 14144 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.129:46509
W20240501 14:15:56.839987 14335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:15:56.840162 14335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:15:56.840201 14335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:15:56.842993 14335 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:15:56.843068 14335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:15:56.843153 14335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.130
I20240501 14:15:56.846522 14335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:34331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.7.39.130
--webserver_port=0
--tserver_master_addrs=127.7.39.190:36791
--never_fsync=true
--heap_profile_path=/tmp/kudu.14335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.130
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:15:56.847258 14335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:15:56.848266 14335 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:15:56.856177 14341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:15:56.856617 14344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:15:56.857364 14342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:15:56.857952 14335 server_base.cc:1034] running on GCE node
I20240501 14:15:56.858266 14335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:15:56.858779 14335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:15:56.859941 14335 hybrid_clock.cc:648] HybridClock initialized: now 1714572956859925 us; error 26 us; skew 500 ppm
I20240501 14:15:56.861629 14335 webserver.cc:460] Webserver started at http://127.7.39.130:39825/ using document root <none> and password file <none>
I20240501 14:15:56.862134 14335 fs_manager.cc:359] Metadata directory not provided
I20240501 14:15:56.862192 14335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:15:56.862378 14335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:15:56.863955 14335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "17062e38120042aaaca05e153b2c6ccf"
format_stamp: "Formatted at 2024-05-01 14:15:56 on dist-test-slave-rnfw"
I20240501 14:15:56.864400 14335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "17062e38120042aaaca05e153b2c6ccf"
format_stamp: "Formatted at 2024-05-01 14:15:56 on dist-test-slave-rnfw"
I20240501 14:15:56.867498 14335 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:15:56.869410 14350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:15:56.870299 14335 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:15:56.870389 14335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "17062e38120042aaaca05e153b2c6ccf"
format_stamp: "Formatted at 2024-05-01 14:15:56 on dist-test-slave-rnfw"
I20240501 14:15:56.870476 14335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:15:56.893429 14335 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:15:56.893946 14335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:15:56.894110 14335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:15:56.894601 14335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:15:56.895407 14335 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:15:56.895459 14335 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:15:56.895505 14335 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:15:56.895539 14335 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:15:56.904109 14335 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.130:43603
I20240501 14:15:56.904136 14463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.130:43603 every 8 connection(s)
I20240501 14:15:56.904958 14335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 14:15:56.913372 14464 heartbeater.cc:347] Connected to a master server at 127.7.39.190:36791
I20240501 14:15:56.913573 14464 heartbeater.cc:464] Registering TS with master...
I20240501 14:15:56.913697 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 14335
I20240501 14:15:56.913827 7326 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 14:15:56.914090 14464 heartbeater.cc:511] Master 127.7.39.190:36791 requested a full tablet report, sending...
I20240501 14:15:56.915807 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.131:0
--local_ip_for_outbound_sockets=127.7.39.131
--webserver_interface=127.7.39.131
--webserver_port=0
--tserver_master_addrs=127.7.39.190:36791
--builtin_ntp_servers=127.7.39.148:34331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:15:56.957672 14143 ts_manager.cc:194] Registered new tserver with Master: 17062e38120042aaaca05e153b2c6ccf (127.7.39.130:43603)
I20240501 14:15:56.958616 14143 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.130:35137
W20240501 14:15:57.024686 14468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:15:57.024854 14468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:15:57.024893 14468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:15:57.027645 14468 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:15:57.027698 14468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:15:57.027757 14468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.131
I20240501 14:15:57.031131 14468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:34331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.7.39.131
--webserver_port=0
--tserver_master_addrs=127.7.39.190:36791
--never_fsync=true
--heap_profile_path=/tmp/kudu.14468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.131
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:15:57.031893 14468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:15:57.032922 14468 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:15:57.040511 14473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:15:57.040858 14474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:15:57.041525 14476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:15:57.042150 14468 server_base.cc:1034] running on GCE node
I20240501 14:15:57.042527 14468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:15:57.043102 14468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:15:57.044262 14468 hybrid_clock.cc:648] HybridClock initialized: now 1714572957044239 us; error 42 us; skew 500 ppm
I20240501 14:15:57.045991 14468 webserver.cc:460] Webserver started at http://127.7.39.131:41067/ using document root <none> and password file <none>
I20240501 14:15:57.046491 14468 fs_manager.cc:359] Metadata directory not provided
I20240501 14:15:57.046553 14468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:15:57.046739 14468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:15:57.048319 14468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "e625ab1b03d14e2aae194052d2c67bc6"
format_stamp: "Formatted at 2024-05-01 14:15:57 on dist-test-slave-rnfw"
I20240501 14:15:57.048800 14468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "e625ab1b03d14e2aae194052d2c67bc6"
format_stamp: "Formatted at 2024-05-01 14:15:57 on dist-test-slave-rnfw"
I20240501 14:15:57.052001 14468 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20240501 14:15:57.054118 14482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:15:57.055016 14468 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:15:57.055156 14468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "e625ab1b03d14e2aae194052d2c67bc6"
format_stamp: "Formatted at 2024-05-01 14:15:57 on dist-test-slave-rnfw"
I20240501 14:15:57.055253 14468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:15:57.077054 14468 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:15:57.077536 14468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:15:57.077673 14468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:15:57.078114 14468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:15:57.078879 14468 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:15:57.078927 14468 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:15:57.078976 14468 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:15:57.079005 14468 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:15:57.086831 14468 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.131:33199
I20240501 14:15:57.086861 14595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.131:33199 every 8 connection(s)
I20240501 14:15:57.087767 14468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 14:15:57.092283 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 14468
I20240501 14:15:57.092398 7326 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 14:15:57.094894 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.132:0
--local_ip_for_outbound_sockets=127.7.39.132
--webserver_interface=127.7.39.132
--webserver_port=0
--tserver_master_addrs=127.7.39.190:36791
--builtin_ntp_servers=127.7.39.148:34331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:15:57.096274 14596 heartbeater.cc:347] Connected to a master server at 127.7.39.190:36791
I20240501 14:15:57.096498 14596 heartbeater.cc:464] Registering TS with master...
I20240501 14:15:57.096949 14596 heartbeater.cc:511] Master 127.7.39.190:36791 requested a full tablet report, sending...
I20240501 14:15:57.131372 14143 ts_manager.cc:194] Registered new tserver with Master: e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131:33199)
I20240501 14:15:57.132190 14143 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.131:50939
W20240501 14:15:57.197213 14599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:15:57.197384 14599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:15:57.197423 14599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:15:57.200147 14599 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:15:57.200204 14599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:15:57.200265 14599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.132
I20240501 14:15:57.203560 14599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:34331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.7.39.132
--webserver_port=0
--tserver_master_addrs=127.7.39.190:36791
--never_fsync=true
--heap_profile_path=/tmp/kudu.14599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.132
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:15:57.204241 14599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:15:57.205178 14599 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:15:57.212839 14605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:15:57.213419 14606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:15:57.214013 14608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:15:57.214860 14599 server_base.cc:1034] running on GCE node
I20240501 14:15:57.215207 14599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:15:57.215708 14599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:15:57.216845 14599 hybrid_clock.cc:648] HybridClock initialized: now 1714572957216835 us; error 26 us; skew 500 ppm
I20240501 14:15:57.218421 14599 webserver.cc:460] Webserver started at http://127.7.39.132:39171/ using document root <none> and password file <none>
I20240501 14:15:57.218955 14599 fs_manager.cc:359] Metadata directory not provided
I20240501 14:15:57.219010 14599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:15:57.219202 14599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:15:57.220742 14599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "e3770e9581534f77a7506253b8d201b0"
format_stamp: "Formatted at 2024-05-01 14:15:57 on dist-test-slave-rnfw"
I20240501 14:15:57.221190 14599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "e3770e9581534f77a7506253b8d201b0"
format_stamp: "Formatted at 2024-05-01 14:15:57 on dist-test-slave-rnfw"
I20240501 14:15:57.224267 14599 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:15:57.226135 14614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:15:57.227027 14599 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:15:57.227165 14599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "e3770e9581534f77a7506253b8d201b0"
format_stamp: "Formatted at 2024-05-01 14:15:57 on dist-test-slave-rnfw"
I20240501 14:15:57.227255 14599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:15:57.235430 14599 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:15:57.235906 14599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:15:57.236032 14599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:15:57.236443 14599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:15:57.237174 14599 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:15:57.237231 14599 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:15:57.237277 14599 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:15:57.237308 14599 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:15:57.245450 14599 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.132:38927
I20240501 14:15:57.245483 14727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.132:38927 every 8 connection(s)
I20240501 14:15:57.246206 14599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 14:15:57.251772 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 14599
I20240501 14:15:57.251942 7326 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 14:15:57.254403 14728 heartbeater.cc:347] Connected to a master server at 127.7.39.190:36791
I20240501 14:15:57.254455 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.133:0
--local_ip_for_outbound_sockets=127.7.39.133
--webserver_interface=127.7.39.133
--webserver_port=0
--tserver_master_addrs=127.7.39.190:36791
--builtin_ntp_servers=127.7.39.148:34331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:15:57.254623 14728 heartbeater.cc:464] Registering TS with master...
I20240501 14:15:57.255273 14728 heartbeater.cc:511] Master 127.7.39.190:36791 requested a full tablet report, sending...
I20240501 14:15:57.289896 14143 ts_manager.cc:194] Registered new tserver with Master: e3770e9581534f77a7506253b8d201b0 (127.7.39.132:38927)
I20240501 14:15:57.290638 14143 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.132:50653
W20240501 14:15:57.356679 14731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:15:57.356891 14731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:15:57.356959 14731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:15:57.359830 14731 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:15:57.359894 14731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:15:57.359977 14731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.133
I20240501 14:15:57.363242 14731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:34331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.7.39.133
--webserver_port=0
--tserver_master_addrs=127.7.39.190:36791
--never_fsync=true
--heap_profile_path=/tmp/kudu.14731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.133
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:15:57.363957 14731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:15:57.364854 14731 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:15:57.372398 14740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:15:57.372870 14737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:15:57.373556 14738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:15:57.374193 14731 server_base.cc:1034] running on GCE node
I20240501 14:15:57.374482 14731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:15:57.374979 14731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:15:57.376127 14731 hybrid_clock.cc:648] HybridClock initialized: now 1714572957376117 us; error 21 us; skew 500 ppm
I20240501 14:15:57.377622 14731 webserver.cc:460] Webserver started at http://127.7.39.133:45277/ using document root <none> and password file <none>
I20240501 14:15:57.378091 14731 fs_manager.cc:359] Metadata directory not provided
I20240501 14:15:57.378145 14731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:15:57.378302 14731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:15:57.379735 14731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "50161459714942cb9b4ae7743f76d07d"
format_stamp: "Formatted at 2024-05-01 14:15:57 on dist-test-slave-rnfw"
I20240501 14:15:57.380167 14731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "50161459714942cb9b4ae7743f76d07d"
format_stamp: "Formatted at 2024-05-01 14:15:57 on dist-test-slave-rnfw"
I20240501 14:15:57.383033 14731 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20240501 14:15:57.384871 14746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:15:57.385718 14731 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240501 14:15:57.385812 14731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "50161459714942cb9b4ae7743f76d07d"
format_stamp: "Formatted at 2024-05-01 14:15:57 on dist-test-slave-rnfw"
I20240501 14:15:57.385885 14731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:15:57.405110 14731 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:15:57.405586 14731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:15:57.405723 14731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:15:57.406234 14731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:15:57.406972 14731 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:15:57.407018 14731 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:15:57.407133 14731 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:15:57.407161 14731 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:15:57.415271 14731 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.133:36729
I20240501 14:15:57.415298 14859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.133:36729 every 8 connection(s)
I20240501 14:15:57.416034 14731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 14:15:57.420213 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 14731
I20240501 14:15:57.420327 7326 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 14:15:57.423949 14860 heartbeater.cc:347] Connected to a master server at 127.7.39.190:36791
I20240501 14:15:57.424113 14860 heartbeater.cc:464] Registering TS with master...
I20240501 14:15:57.424515 14860 heartbeater.cc:511] Master 127.7.39.190:36791 requested a full tablet report, sending...
I20240501 14:15:57.457372 14143 ts_manager.cc:194] Registered new tserver with Master: 50161459714942cb9b4ae7743f76d07d (127.7.39.133:36729)
I20240501 14:15:57.457978 14143 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.133:34619
I20240501 14:15:57.468607 7326 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 14:15:57.476926 7326 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 14:15:57.476995 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 14731
I20240501 14:15:57.482514 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 14599
I20240501 14:15:57.783996 14332 heartbeater.cc:503] Master 127.7.39.190:36791 was elected leader, sending a full tablet report...
I20240501 14:15:57.960965 14464 heartbeater.cc:503] Master 127.7.39.190:36791 was elected leader, sending a full tablet report...
I20240501 14:15:58.134508 14596 heartbeater.cc:503] Master 127.7.39.190:36791 was elected leader, sending a full tablet report...
I20240501 14:16:01.239287 14143 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240501 14:16:01.242612 14143 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 14:16:01.261420 14529 tablet_service.cc:1467] Processing CreateTablet for tablet 3ded831efb0846a192400e8856dbd9b6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=066c15421d0b476bbe4204501666d610]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:01.261437 14266 tablet_service.cc:1467] Processing CreateTablet for tablet 5f2b92ee1a1c46bcb36bc24563886d18 (DEFAULT_TABLE table=rebalance_test_table_0 [id=066c15421d0b476bbe4204501666d610]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:01.261420 14530 tablet_service.cc:1467] Processing CreateTablet for tablet 5f2b92ee1a1c46bcb36bc24563886d18 (DEFAULT_TABLE table=rebalance_test_table_0 [id=066c15421d0b476bbe4204501666d610]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:01.262282 14528 tablet_service.cc:1467] Processing CreateTablet for tablet 84e2150d96c4451d95302894342228e8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=066c15421d0b476bbe4204501666d610]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:01.262420 14266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f2b92ee1a1c46bcb36bc24563886d18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:01.262420 14530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f2b92ee1a1c46bcb36bc24563886d18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:01.262753 14529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ded831efb0846a192400e8856dbd9b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:01.262935 14528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84e2150d96c4451d95302894342228e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:01.265496 14264 tablet_service.cc:1467] Processing CreateTablet for tablet 84e2150d96c4451d95302894342228e8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=066c15421d0b476bbe4204501666d610]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:01.265815 14264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84e2150d96c4451d95302894342228e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:01.267771 14265 tablet_service.cc:1467] Processing CreateTablet for tablet 3ded831efb0846a192400e8856dbd9b6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=066c15421d0b476bbe4204501666d610]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:01.268023 14265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ded831efb0846a192400e8856dbd9b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:01.270539 14879 tablet_bootstrap.cc:492] T 5f2b92ee1a1c46bcb36bc24563886d18 P c87cbbf2d3ab4143815ef2c74ae760db: Bootstrap starting.
I20240501 14:16:01.273142 14879 tablet_bootstrap.cc:654] T 5f2b92ee1a1c46bcb36bc24563886d18 P c87cbbf2d3ab4143815ef2c74ae760db: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:01.273999 14879 log.cc:824] T 5f2b92ee1a1c46bcb36bc24563886d18 P c87cbbf2d3ab4143815ef2c74ae760db: Log is configured to *not* fsync() on all Append() calls
I20240501 14:16:01.274235 14880 tablet_bootstrap.cc:492] T 84e2150d96c4451d95302894342228e8 P e625ab1b03d14e2aae194052d2c67bc6: Bootstrap starting.
I20240501 14:16:01.276552 14880 tablet_bootstrap.cc:654] T 84e2150d96c4451d95302894342228e8 P e625ab1b03d14e2aae194052d2c67bc6: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:01.276928 14879 tablet_bootstrap.cc:492] T 5f2b92ee1a1c46bcb36bc24563886d18 P c87cbbf2d3ab4143815ef2c74ae760db: No bootstrap required, opened a new log
I20240501 14:16:01.277125 14879 ts_tablet_manager.cc:1367] T 5f2b92ee1a1c46bcb36bc24563886d18 P c87cbbf2d3ab4143815ef2c74ae760db: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240501 14:16:01.277000 14398 tablet_service.cc:1467] Processing CreateTablet for tablet 5f2b92ee1a1c46bcb36bc24563886d18 (DEFAULT_TABLE table=rebalance_test_table_0 [id=066c15421d0b476bbe4204501666d610]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:01.277364 14880 log.cc:824] T 84e2150d96c4451d95302894342228e8 P e625ab1b03d14e2aae194052d2c67bc6: Log is configured to *not* fsync() on all Append() calls
I20240501 14:16:01.277920 14398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f2b92ee1a1c46bcb36bc24563886d18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:01.277257 14397 tablet_service.cc:1467] Processing CreateTablet for tablet 3ded831efb0846a192400e8856dbd9b6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=066c15421d0b476bbe4204501666d610]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:01.278499 14397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ded831efb0846a192400e8856dbd9b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:01.279230 14880 tablet_bootstrap.cc:492] T 84e2150d96c4451d95302894342228e8 P e625ab1b03d14e2aae194052d2c67bc6: No bootstrap required, opened a new log
I20240501 14:16:01.279408 14880 ts_tablet_manager.cc:1367] T 84e2150d96c4451d95302894342228e8 P e625ab1b03d14e2aae194052d2c67bc6: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.004s
I20240501 14:16:01.277000 14396 tablet_service.cc:1467] Processing CreateTablet for tablet 84e2150d96c4451d95302894342228e8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=066c15421d0b476bbe4204501666d610]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:01.279672 14396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84e2150d96c4451d95302894342228e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:01.281266 14879 raft_consensus.cc:348] T 5f2b92ee1a1c46bcb36bc24563886d18 P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } }
I20240501 14:16:01.281534 14879 raft_consensus.cc:374] T 5f2b92ee1a1c46bcb36bc24563886d18 P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:01.281615 14879 raft_consensus.cc:729] T 5f2b92ee1a1c46bcb36bc24563886d18 P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c87cbbf2d3ab4143815ef2c74ae760db, State: Initialized, Role: FOLLOWER
I20240501 14:16:01.282070 14879 consensus_queue.cc:260] T 5f2b92ee1a1c46bcb36bc24563886d18 P c87cbbf2d3ab4143815ef2c74ae760db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } }
I20240501 14:16:01.283167 14879 ts_tablet_manager.cc:1398] T 5f2b92ee1a1c46bcb36bc24563886d18 P c87cbbf2d3ab4143815ef2c74ae760db: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240501 14:16:01.283175 14880 raft_consensus.cc:348] T 84e2150d96c4451d95302894342228e8 P e625ab1b03d14e2aae194052d2c67bc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:01.283393 14880 raft_consensus.cc:374] T 84e2150d96c4451d95302894342228e8 P e625ab1b03d14e2aae194052d2c67bc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:01.283475 14880 raft_consensus.cc:729] T 84e2150d96c4451d95302894342228e8 P e625ab1b03d14e2aae194052d2c67bc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e625ab1b03d14e2aae194052d2c67bc6, State: Initialized, Role: FOLLOWER
I20240501 14:16:01.284060 14880 consensus_queue.cc:260] T 84e2150d96c4451d95302894342228e8 P e625ab1b03d14e2aae194052d2c67bc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:01.285542 14880 ts_tablet_manager.cc:1398] T 84e2150d96c4451d95302894342228e8 P e625ab1b03d14e2aae194052d2c67bc6: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240501 14:16:01.285658 14879 tablet_bootstrap.cc:492] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db: Bootstrap starting.
I20240501 14:16:01.285923 14880 tablet_bootstrap.cc:492] T 5f2b92ee1a1c46bcb36bc24563886d18 P e625ab1b03d14e2aae194052d2c67bc6: Bootstrap starting.
I20240501 14:16:01.286753 14880 tablet_bootstrap.cc:654] T 5f2b92ee1a1c46bcb36bc24563886d18 P e625ab1b03d14e2aae194052d2c67bc6: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:01.287504 14879 tablet_bootstrap.cc:654] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:01.288300 14880 tablet_bootstrap.cc:492] T 5f2b92ee1a1c46bcb36bc24563886d18 P e625ab1b03d14e2aae194052d2c67bc6: No bootstrap required, opened a new log
I20240501 14:16:01.288431 14880 ts_tablet_manager.cc:1367] T 5f2b92ee1a1c46bcb36bc24563886d18 P e625ab1b03d14e2aae194052d2c67bc6: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:16:01.289079 14884 tablet_bootstrap.cc:492] T 84e2150d96c4451d95302894342228e8 P 17062e38120042aaaca05e153b2c6ccf: Bootstrap starting.
I20240501 14:16:01.289039 14880 raft_consensus.cc:348] T 5f2b92ee1a1c46bcb36bc24563886d18 P e625ab1b03d14e2aae194052d2c67bc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } }
I20240501 14:16:01.289147 14880 raft_consensus.cc:374] T 5f2b92ee1a1c46bcb36bc24563886d18 P e625ab1b03d14e2aae194052d2c67bc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:01.289181 14880 raft_consensus.cc:729] T 5f2b92ee1a1c46bcb36bc24563886d18 P e625ab1b03d14e2aae194052d2c67bc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e625ab1b03d14e2aae194052d2c67bc6, State: Initialized, Role: FOLLOWER
I20240501 14:16:01.289402 14879 tablet_bootstrap.cc:492] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db: No bootstrap required, opened a new log
I20240501 14:16:01.289350 14880 consensus_queue.cc:260] T 5f2b92ee1a1c46bcb36bc24563886d18 P e625ab1b03d14e2aae194052d2c67bc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } }
I20240501 14:16:01.289489 14879 ts_tablet_manager.cc:1367] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:16:01.289718 14880 ts_tablet_manager.cc:1398] T 5f2b92ee1a1c46bcb36bc24563886d18 P e625ab1b03d14e2aae194052d2c67bc6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:16:01.289903 14880 tablet_bootstrap.cc:492] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6: Bootstrap starting.
I20240501 14:16:01.289940 14879 raft_consensus.cc:348] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:01.290069 14879 raft_consensus.cc:374] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:01.290127 14879 raft_consensus.cc:729] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c87cbbf2d3ab4143815ef2c74ae760db, State: Initialized, Role: FOLLOWER
I20240501 14:16:01.290261 14879 consensus_queue.cc:260] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:01.290582 14879 ts_tablet_manager.cc:1398] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:16:01.290757 14880 tablet_bootstrap.cc:654] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:01.290776 14879 tablet_bootstrap.cc:492] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db: Bootstrap starting.
I20240501 14:16:01.291584 14884 tablet_bootstrap.cc:654] T 84e2150d96c4451d95302894342228e8 P 17062e38120042aaaca05e153b2c6ccf: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:01.291703 14879 tablet_bootstrap.cc:654] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:01.292284 14884 log.cc:824] T 84e2150d96c4451d95302894342228e8 P 17062e38120042aaaca05e153b2c6ccf: Log is configured to *not* fsync() on all Append() calls
I20240501 14:16:01.292912 14879 tablet_bootstrap.cc:492] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db: No bootstrap required, opened a new log
I20240501 14:16:01.293038 14879 ts_tablet_manager.cc:1367] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:16:01.293584 14879 raft_consensus.cc:348] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:01.293704 14879 raft_consensus.cc:374] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:01.293740 14880 tablet_bootstrap.cc:492] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6: No bootstrap required, opened a new log
I20240501 14:16:01.293740 14879 raft_consensus.cc:729] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c87cbbf2d3ab4143815ef2c74ae760db, State: Initialized, Role: FOLLOWER
I20240501 14:16:01.293818 14880 ts_tablet_manager.cc:1367] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:16:01.293859 14879 consensus_queue.cc:260] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:01.294142 14879 ts_tablet_manager.cc:1398] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:16:01.294263 14880 raft_consensus.cc:348] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:01.294356 14880 raft_consensus.cc:374] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:01.294389 14880 raft_consensus.cc:729] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e625ab1b03d14e2aae194052d2c67bc6, State: Initialized, Role: FOLLOWER
I20240501 14:16:01.294515 14880 consensus_queue.cc:260] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:01.294768 14880 ts_tablet_manager.cc:1398] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:16:01.296109 14884 tablet_bootstrap.cc:492] T 84e2150d96c4451d95302894342228e8 P 17062e38120042aaaca05e153b2c6ccf: No bootstrap required, opened a new log
I20240501 14:16:01.296314 14884 ts_tablet_manager.cc:1367] T 84e2150d96c4451d95302894342228e8 P 17062e38120042aaaca05e153b2c6ccf: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240501 14:16:01.299192 14884 raft_consensus.cc:348] T 84e2150d96c4451d95302894342228e8 P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:01.299356 14884 raft_consensus.cc:374] T 84e2150d96c4451d95302894342228e8 P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:01.299403 14884 raft_consensus.cc:729] T 84e2150d96c4451d95302894342228e8 P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17062e38120042aaaca05e153b2c6ccf, State: Initialized, Role: FOLLOWER
I20240501 14:16:01.299741 14884 consensus_queue.cc:260] T 84e2150d96c4451d95302894342228e8 P 17062e38120042aaaca05e153b2c6ccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:01.300459 14884 ts_tablet_manager.cc:1398] T 84e2150d96c4451d95302894342228e8 P 17062e38120042aaaca05e153b2c6ccf: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 14:16:01.300761 14884 tablet_bootstrap.cc:492] T 3ded831efb0846a192400e8856dbd9b6 P 17062e38120042aaaca05e153b2c6ccf: Bootstrap starting.
I20240501 14:16:01.301716 14884 tablet_bootstrap.cc:654] T 3ded831efb0846a192400e8856dbd9b6 P 17062e38120042aaaca05e153b2c6ccf: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:01.302860 14884 tablet_bootstrap.cc:492] T 3ded831efb0846a192400e8856dbd9b6 P 17062e38120042aaaca05e153b2c6ccf: No bootstrap required, opened a new log
I20240501 14:16:01.302969 14884 ts_tablet_manager.cc:1367] T 3ded831efb0846a192400e8856dbd9b6 P 17062e38120042aaaca05e153b2c6ccf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:16:01.303496 14884 raft_consensus.cc:348] T 3ded831efb0846a192400e8856dbd9b6 P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:01.303582 14884 raft_consensus.cc:374] T 3ded831efb0846a192400e8856dbd9b6 P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:01.303603 14884 raft_consensus.cc:729] T 3ded831efb0846a192400e8856dbd9b6 P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17062e38120042aaaca05e153b2c6ccf, State: Initialized, Role: FOLLOWER
I20240501 14:16:01.303712 14884 consensus_queue.cc:260] T 3ded831efb0846a192400e8856dbd9b6 P 17062e38120042aaaca05e153b2c6ccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:01.303941 14884 ts_tablet_manager.cc:1398] T 3ded831efb0846a192400e8856dbd9b6 P 17062e38120042aaaca05e153b2c6ccf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:16:01.304119 14884 tablet_bootstrap.cc:492] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf: Bootstrap starting.
I20240501 14:16:01.304929 14884 tablet_bootstrap.cc:654] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:01.306041 14884 tablet_bootstrap.cc:492] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf: No bootstrap required, opened a new log
I20240501 14:16:01.306143 14884 ts_tablet_manager.cc:1367] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:16:01.306632 14884 raft_consensus.cc:348] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } }
I20240501 14:16:01.306715 14884 raft_consensus.cc:374] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:01.306736 14884 raft_consensus.cc:729] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17062e38120042aaaca05e153b2c6ccf, State: Initialized, Role: FOLLOWER
I20240501 14:16:01.306823 14884 consensus_queue.cc:260] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } }
I20240501 14:16:01.307018 14884 ts_tablet_manager.cc:1398] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:16:01.321588 14883 raft_consensus.cc:482] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:16:01.321769 14883 raft_consensus.cc:504] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:01.322815 14883 leader_election.cc:288] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17062e38120042aaaca05e153b2c6ccf (127.7.39.130:43603), e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131:33199)
I20240501 14:16:01.328022 14418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84e2150d96c4451d95302894342228e8" candidate_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17062e38120042aaaca05e153b2c6ccf" is_pre_election: true
I20240501 14:16:01.328281 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84e2150d96c4451d95302894342228e8" candidate_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e625ab1b03d14e2aae194052d2c67bc6" is_pre_election: true
I20240501 14:16:01.328382 14418 raft_consensus.cc:2454] T 84e2150d96c4451d95302894342228e8 P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c87cbbf2d3ab4143815ef2c74ae760db in term 0.
I20240501 14:16:01.328570 14550 raft_consensus.cc:2454] T 84e2150d96c4451d95302894342228e8 P e625ab1b03d14e2aae194052d2c67bc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c87cbbf2d3ab4143815ef2c74ae760db in term 0.
I20240501 14:16:01.328860 14220 leader_election.cc:302] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17062e38120042aaaca05e153b2c6ccf, c87cbbf2d3ab4143815ef2c74ae760db; no voters:
I20240501 14:16:01.329134 14883 raft_consensus.cc:2781] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:16:01.329209 14883 raft_consensus.cc:482] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:16:01.329241 14883 raft_consensus.cc:3037] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:01.330166 14883 raft_consensus.cc:504] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:01.330520 14883 leader_election.cc:288] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db [CANDIDATE]: Term 1 election: Requested vote from peers 17062e38120042aaaca05e153b2c6ccf (127.7.39.130:43603), e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131:33199)
I20240501 14:16:01.330916 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84e2150d96c4451d95302894342228e8" candidate_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e625ab1b03d14e2aae194052d2c67bc6"
I20240501 14:16:01.330917 14418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84e2150d96c4451d95302894342228e8" candidate_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17062e38120042aaaca05e153b2c6ccf"
I20240501 14:16:01.331063 14550 raft_consensus.cc:3037] T 84e2150d96c4451d95302894342228e8 P e625ab1b03d14e2aae194052d2c67bc6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:01.331068 14418 raft_consensus.cc:3037] T 84e2150d96c4451d95302894342228e8 P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:01.332178 14550 raft_consensus.cc:2454] T 84e2150d96c4451d95302894342228e8 P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c87cbbf2d3ab4143815ef2c74ae760db in term 1.
I20240501 14:16:01.332489 14222 leader_election.cc:302] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c87cbbf2d3ab4143815ef2c74ae760db, e625ab1b03d14e2aae194052d2c67bc6; no voters:
I20240501 14:16:01.332702 14883 raft_consensus.cc:2781] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:16:01.332757 14418 raft_consensus.cc:2454] T 84e2150d96c4451d95302894342228e8 P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c87cbbf2d3ab4143815ef2c74ae760db in term 1.
I20240501 14:16:01.332975 14883 raft_consensus.cc:686] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 LEADER]: Becoming Leader. State: Replica: c87cbbf2d3ab4143815ef2c74ae760db, State: Running, Role: LEADER
I20240501 14:16:01.333343 14883 consensus_queue.cc:237] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:01.336385 14143 catalog_manager.cc:5484] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db reported cstate change: term changed from 0 to 1, leader changed from <none> to c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129). New cstate: current_term: 1 leader_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } health_report { overall_health: UNKNOWN } } }
W20240501 14:16:01.341534 14597 tablet.cc:2369] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:16:01.388490 14892 raft_consensus.cc:482] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:16:01.388655 14892 raft_consensus.cc:504] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } }
I20240501 14:16:01.389812 14892 leader_election.cc:288] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131:33199), c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129:42369)
I20240501 14:16:01.394568 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f2b92ee1a1c46bcb36bc24563886d18" candidate_uuid: "17062e38120042aaaca05e153b2c6ccf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e625ab1b03d14e2aae194052d2c67bc6" is_pre_election: true
I20240501 14:16:01.394724 14550 raft_consensus.cc:2454] T 5f2b92ee1a1c46bcb36bc24563886d18 P e625ab1b03d14e2aae194052d2c67bc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17062e38120042aaaca05e153b2c6ccf in term 0.
I20240501 14:16:01.394681 14286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f2b92ee1a1c46bcb36bc24563886d18" candidate_uuid: "17062e38120042aaaca05e153b2c6ccf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" is_pre_election: true
I20240501 14:16:01.394902 14286 raft_consensus.cc:2454] T 5f2b92ee1a1c46bcb36bc24563886d18 P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17062e38120042aaaca05e153b2c6ccf in term 0.
I20240501 14:16:01.395102 14354 leader_election.cc:302] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17062e38120042aaaca05e153b2c6ccf, e625ab1b03d14e2aae194052d2c67bc6; no voters:
I20240501 14:16:01.395372 14892 raft_consensus.cc:2781] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:16:01.395447 14892 raft_consensus.cc:482] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:16:01.395498 14892 raft_consensus.cc:3037] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:01.396379 14892 raft_consensus.cc:504] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } }
I20240501 14:16:01.396729 14892 leader_election.cc:288] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf [CANDIDATE]: Term 1 election: Requested vote from peers e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131:33199), c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129:42369)
I20240501 14:16:01.397091 14286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f2b92ee1a1c46bcb36bc24563886d18" candidate_uuid: "17062e38120042aaaca05e153b2c6ccf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c87cbbf2d3ab4143815ef2c74ae760db"
I20240501 14:16:01.397091 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f2b92ee1a1c46bcb36bc24563886d18" candidate_uuid: "17062e38120042aaaca05e153b2c6ccf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e625ab1b03d14e2aae194052d2c67bc6"
I20240501 14:16:01.397205 14286 raft_consensus.cc:3037] T 5f2b92ee1a1c46bcb36bc24563886d18 P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:01.397205 14550 raft_consensus.cc:3037] T 5f2b92ee1a1c46bcb36bc24563886d18 P e625ab1b03d14e2aae194052d2c67bc6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:01.398181 14286 raft_consensus.cc:2454] T 5f2b92ee1a1c46bcb36bc24563886d18 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17062e38120042aaaca05e153b2c6ccf in term 1.
I20240501 14:16:01.398181 14550 raft_consensus.cc:2454] T 5f2b92ee1a1c46bcb36bc24563886d18 P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17062e38120042aaaca05e153b2c6ccf in term 1.
I20240501 14:16:01.398481 14354 leader_election.cc:302] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17062e38120042aaaca05e153b2c6ccf, e625ab1b03d14e2aae194052d2c67bc6; no voters:
I20240501 14:16:01.398685 14892 raft_consensus.cc:2781] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:16:01.398937 14892 raft_consensus.cc:686] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf [term 1 LEADER]: Becoming Leader. State: Replica: 17062e38120042aaaca05e153b2c6ccf, State: Running, Role: LEADER
I20240501 14:16:01.399262 14892 consensus_queue.cc:237] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } }
I20240501 14:16:01.402201 14143 catalog_manager.cc:5484] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf reported cstate change: term changed from 0 to 1, leader changed from <none> to 17062e38120042aaaca05e153b2c6ccf (127.7.39.130). New cstate: current_term: 1 leader_uuid: "17062e38120042aaaca05e153b2c6ccf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } health_report { overall_health: UNKNOWN } } }
W20240501 14:16:01.409026 14465 tablet.cc:2369] T 3ded831efb0846a192400e8856dbd9b6 P 17062e38120042aaaca05e153b2c6ccf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:16:01.488988 14333 tablet.cc:2369] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:16:01.531306 14883 raft_consensus.cc:482] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:16:01.531445 14883 raft_consensus.cc:504] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:01.531854 14883 leader_election.cc:288] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17062e38120042aaaca05e153b2c6ccf (127.7.39.130:43603), e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131:33199)
I20240501 14:16:01.532423 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ded831efb0846a192400e8856dbd9b6" candidate_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e625ab1b03d14e2aae194052d2c67bc6" is_pre_election: true
I20240501 14:16:01.532442 14418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ded831efb0846a192400e8856dbd9b6" candidate_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17062e38120042aaaca05e153b2c6ccf" is_pre_election: true
I20240501 14:16:01.532584 14418 raft_consensus.cc:2454] T 3ded831efb0846a192400e8856dbd9b6 P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c87cbbf2d3ab4143815ef2c74ae760db in term 0.
I20240501 14:16:01.532613 14550 raft_consensus.cc:2454] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c87cbbf2d3ab4143815ef2c74ae760db in term 0.
I20240501 14:16:01.532928 14220 leader_election.cc:302] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17062e38120042aaaca05e153b2c6ccf, c87cbbf2d3ab4143815ef2c74ae760db; no voters:
I20240501 14:16:01.533142 14883 raft_consensus.cc:2781] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:16:01.533216 14883 raft_consensus.cc:482] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:16:01.533275 14883 raft_consensus.cc:3037] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:01.534303 14883 raft_consensus.cc:504] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:01.534718 14883 leader_election.cc:288] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db [CANDIDATE]: Term 1 election: Requested vote from peers 17062e38120042aaaca05e153b2c6ccf (127.7.39.130:43603), e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131:33199)
I20240501 14:16:01.535187 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ded831efb0846a192400e8856dbd9b6" candidate_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e625ab1b03d14e2aae194052d2c67bc6"
I20240501 14:16:01.535184 14418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ded831efb0846a192400e8856dbd9b6" candidate_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17062e38120042aaaca05e153b2c6ccf"
I20240501 14:16:01.535290 14550 raft_consensus.cc:3037] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:01.535316 14418 raft_consensus.cc:3037] T 3ded831efb0846a192400e8856dbd9b6 P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:01.536036 14550 raft_consensus.cc:2454] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c87cbbf2d3ab4143815ef2c74ae760db in term 1.
I20240501 14:16:01.536128 14418 raft_consensus.cc:2454] T 3ded831efb0846a192400e8856dbd9b6 P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c87cbbf2d3ab4143815ef2c74ae760db in term 1.
I20240501 14:16:01.536281 14222 leader_election.cc:302] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c87cbbf2d3ab4143815ef2c74ae760db, e625ab1b03d14e2aae194052d2c67bc6; no voters:
I20240501 14:16:01.536448 14883 raft_consensus.cc:2781] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:16:01.536545 14883 raft_consensus.cc:686] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 LEADER]: Becoming Leader. State: Replica: c87cbbf2d3ab4143815ef2c74ae760db, State: Running, Role: LEADER
I20240501 14:16:01.536676 14883 consensus_queue.cc:237] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:01.538235 14143 catalog_manager.cc:5484] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db reported cstate change: term changed from 0 to 1, leader changed from <none> to c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129). New cstate: current_term: 1 leader_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } health_report { overall_health: UNKNOWN } } }
W20240501 14:16:01.645578 14904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:16:01.645803 14904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:16:01.648532 14904 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240501 14:16:01.725736 14418 raft_consensus.cc:1261] T 3ded831efb0846a192400e8856dbd9b6 P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Refusing update from remote peer c87cbbf2d3ab4143815ef2c74ae760db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:16:01.725733 14550 raft_consensus.cc:1261] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Refusing update from remote peer c87cbbf2d3ab4143815ef2c74ae760db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:16:01.726320 14897 consensus_queue.cc:1035] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db [LEADER]: Connected to new peer: Peer: permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:16:01.725739 14549 raft_consensus.cc:1261] T 84e2150d96c4451d95302894342228e8 P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Refusing update from remote peer c87cbbf2d3ab4143815ef2c74ae760db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:16:01.726801 14897 consensus_queue.cc:1035] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db [LEADER]: Connected to new peer: Peer: permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:16:01.727373 14286 raft_consensus.cc:1261] T 5f2b92ee1a1c46bcb36bc24563886d18 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Refusing update from remote peer 17062e38120042aaaca05e153b2c6ccf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:16:01.727397 14883 consensus_queue.cc:1035] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db [LEADER]: Connected to new peer: Peer: permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:16:01.728062 14901 consensus_queue.cc:1035] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:16:01.728550 14550 raft_consensus.cc:1261] T 5f2b92ee1a1c46bcb36bc24563886d18 P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Refusing update from remote peer 17062e38120042aaaca05e153b2c6ccf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:16:01.728976 14892 consensus_queue.cc:1035] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:16:01.743211 14940 mvcc.cc:205] Tried to move back new op lower bound from 7022890851229323264 to 7022890849893777408. Current Snapshot: MvccSnapshot[applied={T|T < 7022890851229323264}]
I20240501 14:16:01.743280 14417 raft_consensus.cc:1261] T 84e2150d96c4451d95302894342228e8 P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Refusing update from remote peer c87cbbf2d3ab4143815ef2c74ae760db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:16:01.744136 14936 mvcc.cc:205] Tried to move back new op lower bound from 7022890851262488576 to 7022890849893777408. Current Snapshot: MvccSnapshot[applied={T|T < 7022890851229323264}]
I20240501 14:16:01.743674 14883 consensus_queue.cc:1035] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db [LEADER]: Connected to new peer: Peer: permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:16:01.748669 14939 mvcc.cc:205] Tried to move back new op lower bound from 7022890851229323264 to 7022890849893777408. Current Snapshot: MvccSnapshot[applied={T|T < 7022890851229323264}]
INSERT report
rows total: 2000
time total: 106.935 ms
time per row: 0.0534676 ms
I20240501 14:16:01.798565 14140 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240501 14:16:01.799142 14140 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 14:16:01.804116 14398 tablet_service.cc:1467] Processing CreateTablet for tablet ca266243bcb84e8c9b65d1b096b5725d (DEFAULT_TABLE table=rebalance_test_table_1 [id=2938421fb092484186ed564e78a8fd87]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:01.804117 14529 tablet_service.cc:1467] Processing CreateTablet for tablet ca266243bcb84e8c9b65d1b096b5725d (DEFAULT_TABLE table=rebalance_test_table_1 [id=2938421fb092484186ed564e78a8fd87]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:01.804423 14529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca266243bcb84e8c9b65d1b096b5725d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:01.804420 14398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca266243bcb84e8c9b65d1b096b5725d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:01.804539 14530 tablet_service.cc:1467] Processing CreateTablet for tablet 40732c47173c429aa69c6a41475849de (DEFAULT_TABLE table=rebalance_test_table_1 [id=2938421fb092484186ed564e78a8fd87]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:01.804773 14530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40732c47173c429aa69c6a41475849de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:01.804709 14264 tablet_service.cc:1467] Processing CreateTablet for tablet ca266243bcb84e8c9b65d1b096b5725d (DEFAULT_TABLE table=rebalance_test_table_1 [id=2938421fb092484186ed564e78a8fd87]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:01.804955 14264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca266243bcb84e8c9b65d1b096b5725d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:01.806632 14266 tablet_service.cc:1467] Processing CreateTablet for tablet 40732c47173c429aa69c6a41475849de (DEFAULT_TABLE table=rebalance_test_table_1 [id=2938421fb092484186ed564e78a8fd87]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:01.807081 14884 tablet_bootstrap.cc:492] T ca266243bcb84e8c9b65d1b096b5725d P 17062e38120042aaaca05e153b2c6ccf: Bootstrap starting.
I20240501 14:16:01.807264 14398 tablet_service.cc:1467] Processing CreateTablet for tablet 40732c47173c429aa69c6a41475849de (DEFAULT_TABLE table=rebalance_test_table_1 [id=2938421fb092484186ed564e78a8fd87]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:01.807411 14960 tablet_bootstrap.cc:492] T ca266243bcb84e8c9b65d1b096b5725d P e625ab1b03d14e2aae194052d2c67bc6: Bootstrap starting.
I20240501 14:16:01.807484 14398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40732c47173c429aa69c6a41475849de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:01.807391 14530 tablet_service.cc:1467] Processing CreateTablet for tablet 8ebd7203c0914f6c8e26ab4d20074765 (DEFAULT_TABLE table=rebalance_test_table_1 [id=2938421fb092484186ed564e78a8fd87]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:01.807636 14530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ebd7203c0914f6c8e26ab4d20074765. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:01.807942 14884 tablet_bootstrap.cc:654] T ca266243bcb84e8c9b65d1b096b5725d P 17062e38120042aaaca05e153b2c6ccf: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:01.808210 14960 tablet_bootstrap.cc:654] T ca266243bcb84e8c9b65d1b096b5725d P e625ab1b03d14e2aae194052d2c67bc6: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:01.808776 14266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40732c47173c429aa69c6a41475849de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:01.809194 14884 tablet_bootstrap.cc:492] T ca266243bcb84e8c9b65d1b096b5725d P 17062e38120042aaaca05e153b2c6ccf: No bootstrap required, opened a new log
I20240501 14:16:01.809268 14884 ts_tablet_manager.cc:1367] T ca266243bcb84e8c9b65d1b096b5725d P 17062e38120042aaaca05e153b2c6ccf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:16:01.809737 14884 raft_consensus.cc:348] T ca266243bcb84e8c9b65d1b096b5725d P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:01.809830 14884 raft_consensus.cc:374] T ca266243bcb84e8c9b65d1b096b5725d P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:01.809861 14884 raft_consensus.cc:729] T ca266243bcb84e8c9b65d1b096b5725d P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17062e38120042aaaca05e153b2c6ccf, State: Initialized, Role: FOLLOWER
I20240501 14:16:01.809976 14884 consensus_queue.cc:260] T ca266243bcb84e8c9b65d1b096b5725d P 17062e38120042aaaca05e153b2c6ccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:01.810020 14960 tablet_bootstrap.cc:492] T ca266243bcb84e8c9b65d1b096b5725d P e625ab1b03d14e2aae194052d2c67bc6: No bootstrap required, opened a new log
I20240501 14:16:01.810088 14960 ts_tablet_manager.cc:1367] T ca266243bcb84e8c9b65d1b096b5725d P e625ab1b03d14e2aae194052d2c67bc6: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:16:01.810206 14884 ts_tablet_manager.cc:1398] T ca266243bcb84e8c9b65d1b096b5725d P 17062e38120042aaaca05e153b2c6ccf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:16:01.810333 14884 tablet_bootstrap.cc:492] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf: Bootstrap starting.
I20240501 14:16:01.810482 14960 raft_consensus.cc:348] T ca266243bcb84e8c9b65d1b096b5725d P e625ab1b03d14e2aae194052d2c67bc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:01.810575 14960 raft_consensus.cc:374] T ca266243bcb84e8c9b65d1b096b5725d P e625ab1b03d14e2aae194052d2c67bc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:01.810612 14960 raft_consensus.cc:729] T ca266243bcb84e8c9b65d1b096b5725d P e625ab1b03d14e2aae194052d2c67bc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e625ab1b03d14e2aae194052d2c67bc6, State: Initialized, Role: FOLLOWER
I20240501 14:16:01.810747 14960 consensus_queue.cc:260] T ca266243bcb84e8c9b65d1b096b5725d P e625ab1b03d14e2aae194052d2c67bc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:01.810925 14266 tablet_service.cc:1467] Processing CreateTablet for tablet 8ebd7203c0914f6c8e26ab4d20074765 (DEFAULT_TABLE table=rebalance_test_table_1 [id=2938421fb092484186ed564e78a8fd87]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:01.811031 14960 ts_tablet_manager.cc:1398] T ca266243bcb84e8c9b65d1b096b5725d P e625ab1b03d14e2aae194052d2c67bc6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:16:01.811214 14884 tablet_bootstrap.cc:654] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:01.811228 14266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ebd7203c0914f6c8e26ab4d20074765. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:01.811491 14961 tablet_bootstrap.cc:492] T ca266243bcb84e8c9b65d1b096b5725d P c87cbbf2d3ab4143815ef2c74ae760db: Bootstrap starting.
I20240501 14:16:01.811789 14960 tablet_bootstrap.cc:492] T 40732c47173c429aa69c6a41475849de P e625ab1b03d14e2aae194052d2c67bc6: Bootstrap starting.
I20240501 14:16:01.807264 14397 tablet_service.cc:1467] Processing CreateTablet for tablet 8ebd7203c0914f6c8e26ab4d20074765 (DEFAULT_TABLE table=rebalance_test_table_1 [id=2938421fb092484186ed564e78a8fd87]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:01.812325 14397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ebd7203c0914f6c8e26ab4d20074765. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:01.812987 14960 tablet_bootstrap.cc:654] T 40732c47173c429aa69c6a41475849de P e625ab1b03d14e2aae194052d2c67bc6: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:01.814083 14960 tablet_bootstrap.cc:492] T 40732c47173c429aa69c6a41475849de P e625ab1b03d14e2aae194052d2c67bc6: No bootstrap required, opened a new log
I20240501 14:16:01.814152 14960 ts_tablet_manager.cc:1367] T 40732c47173c429aa69c6a41475849de P e625ab1b03d14e2aae194052d2c67bc6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:16:01.814599 14960 raft_consensus.cc:348] T 40732c47173c429aa69c6a41475849de P e625ab1b03d14e2aae194052d2c67bc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:01.814705 14960 raft_consensus.cc:374] T 40732c47173c429aa69c6a41475849de P e625ab1b03d14e2aae194052d2c67bc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:01.814736 14960 raft_consensus.cc:729] T 40732c47173c429aa69c6a41475849de P e625ab1b03d14e2aae194052d2c67bc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e625ab1b03d14e2aae194052d2c67bc6, State: Initialized, Role: FOLLOWER
I20240501 14:16:01.814849 14960 consensus_queue.cc:260] T 40732c47173c429aa69c6a41475849de P e625ab1b03d14e2aae194052d2c67bc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:01.815034 14961 tablet_bootstrap.cc:654] T ca266243bcb84e8c9b65d1b096b5725d P c87cbbf2d3ab4143815ef2c74ae760db: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:01.815167 14884 tablet_bootstrap.cc:492] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf: No bootstrap required, opened a new log
I20240501 14:16:01.815250 14884 ts_tablet_manager.cc:1367] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240501 14:16:01.815411 14960 ts_tablet_manager.cc:1398] T 40732c47173c429aa69c6a41475849de P e625ab1b03d14e2aae194052d2c67bc6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:16:01.815593 14960 tablet_bootstrap.cc:492] T 8ebd7203c0914f6c8e26ab4d20074765 P e625ab1b03d14e2aae194052d2c67bc6: Bootstrap starting.
I20240501 14:16:01.815665 14884 raft_consensus.cc:348] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:01.815766 14884 raft_consensus.cc:374] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:01.815799 14884 raft_consensus.cc:729] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17062e38120042aaaca05e153b2c6ccf, State: Initialized, Role: FOLLOWER
I20240501 14:16:01.815928 14884 consensus_queue.cc:260] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:01.816192 14884 ts_tablet_manager.cc:1398] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:16:01.816337 14884 tablet_bootstrap.cc:492] T 8ebd7203c0914f6c8e26ab4d20074765 P 17062e38120042aaaca05e153b2c6ccf: Bootstrap starting.
I20240501 14:16:01.816465 14960 tablet_bootstrap.cc:654] T 8ebd7203c0914f6c8e26ab4d20074765 P e625ab1b03d14e2aae194052d2c67bc6: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:01.817164 14884 tablet_bootstrap.cc:654] T 8ebd7203c0914f6c8e26ab4d20074765 P 17062e38120042aaaca05e153b2c6ccf: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:01.817373 14961 tablet_bootstrap.cc:492] T ca266243bcb84e8c9b65d1b096b5725d P c87cbbf2d3ab4143815ef2c74ae760db: No bootstrap required, opened a new log
I20240501 14:16:01.817452 14961 ts_tablet_manager.cc:1367] T ca266243bcb84e8c9b65d1b096b5725d P c87cbbf2d3ab4143815ef2c74ae760db: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20240501 14:16:01.817629 14960 tablet_bootstrap.cc:492] T 8ebd7203c0914f6c8e26ab4d20074765 P e625ab1b03d14e2aae194052d2c67bc6: No bootstrap required, opened a new log
I20240501 14:16:01.817693 14960 ts_tablet_manager.cc:1367] T 8ebd7203c0914f6c8e26ab4d20074765 P e625ab1b03d14e2aae194052d2c67bc6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:16:01.817864 14961 raft_consensus.cc:348] T ca266243bcb84e8c9b65d1b096b5725d P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:01.817960 14961 raft_consensus.cc:374] T ca266243bcb84e8c9b65d1b096b5725d P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:01.817991 14961 raft_consensus.cc:729] T ca266243bcb84e8c9b65d1b096b5725d P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c87cbbf2d3ab4143815ef2c74ae760db, State: Initialized, Role: FOLLOWER
I20240501 14:16:01.818097 14960 raft_consensus.cc:348] T 8ebd7203c0914f6c8e26ab4d20074765 P e625ab1b03d14e2aae194052d2c67bc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:01.818125 14961 consensus_queue.cc:260] T ca266243bcb84e8c9b65d1b096b5725d P c87cbbf2d3ab4143815ef2c74ae760db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:01.818182 14960 raft_consensus.cc:374] T 8ebd7203c0914f6c8e26ab4d20074765 P e625ab1b03d14e2aae194052d2c67bc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:01.818213 14960 raft_consensus.cc:729] T 8ebd7203c0914f6c8e26ab4d20074765 P e625ab1b03d14e2aae194052d2c67bc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e625ab1b03d14e2aae194052d2c67bc6, State: Initialized, Role: FOLLOWER
I20240501 14:16:01.818328 14960 consensus_queue.cc:260] T 8ebd7203c0914f6c8e26ab4d20074765 P e625ab1b03d14e2aae194052d2c67bc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:01.818406 14961 ts_tablet_manager.cc:1398] T ca266243bcb84e8c9b65d1b096b5725d P c87cbbf2d3ab4143815ef2c74ae760db: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:16:01.818558 14961 tablet_bootstrap.cc:492] T 40732c47173c429aa69c6a41475849de P c87cbbf2d3ab4143815ef2c74ae760db: Bootstrap starting.
I20240501 14:16:01.818552 14960 ts_tablet_manager.cc:1398] T 8ebd7203c0914f6c8e26ab4d20074765 P e625ab1b03d14e2aae194052d2c67bc6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:16:01.818876 14884 tablet_bootstrap.cc:492] T 8ebd7203c0914f6c8e26ab4d20074765 P 17062e38120042aaaca05e153b2c6ccf: No bootstrap required, opened a new log
I20240501 14:16:01.818975 14884 ts_tablet_manager.cc:1367] T 8ebd7203c0914f6c8e26ab4d20074765 P 17062e38120042aaaca05e153b2c6ccf: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:16:01.819491 14884 raft_consensus.cc:348] T 8ebd7203c0914f6c8e26ab4d20074765 P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:01.819598 14884 raft_consensus.cc:374] T 8ebd7203c0914f6c8e26ab4d20074765 P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:01.819635 14884 raft_consensus.cc:729] T 8ebd7203c0914f6c8e26ab4d20074765 P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17062e38120042aaaca05e153b2c6ccf, State: Initialized, Role: FOLLOWER
I20240501 14:16:01.819743 14884 consensus_queue.cc:260] T 8ebd7203c0914f6c8e26ab4d20074765 P 17062e38120042aaaca05e153b2c6ccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:01.819818 14961 tablet_bootstrap.cc:654] T 40732c47173c429aa69c6a41475849de P c87cbbf2d3ab4143815ef2c74ae760db: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:01.819998 14884 ts_tablet_manager.cc:1398] T 8ebd7203c0914f6c8e26ab4d20074765 P 17062e38120042aaaca05e153b2c6ccf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:16:01.822032 14961 tablet_bootstrap.cc:492] T 40732c47173c429aa69c6a41475849de P c87cbbf2d3ab4143815ef2c74ae760db: No bootstrap required, opened a new log
I20240501 14:16:01.822113 14961 ts_tablet_manager.cc:1367] T 40732c47173c429aa69c6a41475849de P c87cbbf2d3ab4143815ef2c74ae760db: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:16:01.822489 14961 raft_consensus.cc:348] T 40732c47173c429aa69c6a41475849de P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:01.822558 14961 raft_consensus.cc:374] T 40732c47173c429aa69c6a41475849de P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:01.822578 14961 raft_consensus.cc:729] T 40732c47173c429aa69c6a41475849de P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c87cbbf2d3ab4143815ef2c74ae760db, State: Initialized, Role: FOLLOWER
I20240501 14:16:01.822662 14961 consensus_queue.cc:260] T 40732c47173c429aa69c6a41475849de P c87cbbf2d3ab4143815ef2c74ae760db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:01.822844 14961 ts_tablet_manager.cc:1398] T 40732c47173c429aa69c6a41475849de P c87cbbf2d3ab4143815ef2c74ae760db: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:16:01.822952 14961 tablet_bootstrap.cc:492] T 8ebd7203c0914f6c8e26ab4d20074765 P c87cbbf2d3ab4143815ef2c74ae760db: Bootstrap starting.
I20240501 14:16:01.823676 14961 tablet_bootstrap.cc:654] T 8ebd7203c0914f6c8e26ab4d20074765 P c87cbbf2d3ab4143815ef2c74ae760db: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:01.824666 14961 tablet_bootstrap.cc:492] T 8ebd7203c0914f6c8e26ab4d20074765 P c87cbbf2d3ab4143815ef2c74ae760db: No bootstrap required, opened a new log
I20240501 14:16:01.824745 14961 ts_tablet_manager.cc:1367] T 8ebd7203c0914f6c8e26ab4d20074765 P c87cbbf2d3ab4143815ef2c74ae760db: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:16:01.825115 14961 raft_consensus.cc:348] T 8ebd7203c0914f6c8e26ab4d20074765 P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:01.825215 14961 raft_consensus.cc:374] T 8ebd7203c0914f6c8e26ab4d20074765 P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:01.825258 14961 raft_consensus.cc:729] T 8ebd7203c0914f6c8e26ab4d20074765 P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c87cbbf2d3ab4143815ef2c74ae760db, State: Initialized, Role: FOLLOWER
I20240501 14:16:01.825389 14961 consensus_queue.cc:260] T 8ebd7203c0914f6c8e26ab4d20074765 P c87cbbf2d3ab4143815ef2c74ae760db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:01.825632 14961 ts_tablet_manager.cc:1398] T 8ebd7203c0914f6c8e26ab4d20074765 P c87cbbf2d3ab4143815ef2c74ae760db: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:16:01.842978 14951 raft_consensus.cc:482] T ca266243bcb84e8c9b65d1b096b5725d P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:16:01.843137 14951 raft_consensus.cc:504] T ca266243bcb84e8c9b65d1b096b5725d P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:01.843561 14951 leader_election.cc:288] T ca266243bcb84e8c9b65d1b096b5725d P c87cbbf2d3ab4143815ef2c74ae760db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17062e38120042aaaca05e153b2c6ccf (127.7.39.130:43603), e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131:33199)
I20240501 14:16:01.843982 14418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca266243bcb84e8c9b65d1b096b5725d" candidate_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17062e38120042aaaca05e153b2c6ccf" is_pre_election: true
I20240501 14:16:01.843978 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca266243bcb84e8c9b65d1b096b5725d" candidate_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e625ab1b03d14e2aae194052d2c67bc6" is_pre_election: true
I20240501 14:16:01.844105 14418 raft_consensus.cc:2454] T ca266243bcb84e8c9b65d1b096b5725d P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c87cbbf2d3ab4143815ef2c74ae760db in term 0.
I20240501 14:16:01.844122 14549 raft_consensus.cc:2454] T ca266243bcb84e8c9b65d1b096b5725d P e625ab1b03d14e2aae194052d2c67bc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c87cbbf2d3ab4143815ef2c74ae760db in term 0.
I20240501 14:16:01.844393 14220 leader_election.cc:302] T ca266243bcb84e8c9b65d1b096b5725d P c87cbbf2d3ab4143815ef2c74ae760db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17062e38120042aaaca05e153b2c6ccf, c87cbbf2d3ab4143815ef2c74ae760db; no voters:
I20240501 14:16:01.844568 14951 raft_consensus.cc:2781] T ca266243bcb84e8c9b65d1b096b5725d P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:16:01.844640 14951 raft_consensus.cc:482] T ca266243bcb84e8c9b65d1b096b5725d P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:16:01.844695 14951 raft_consensus.cc:3037] T ca266243bcb84e8c9b65d1b096b5725d P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:01.845541 14951 raft_consensus.cc:504] T ca266243bcb84e8c9b65d1b096b5725d P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:01.845825 14951 leader_election.cc:288] T ca266243bcb84e8c9b65d1b096b5725d P c87cbbf2d3ab4143815ef2c74ae760db [CANDIDATE]: Term 1 election: Requested vote from peers 17062e38120042aaaca05e153b2c6ccf (127.7.39.130:43603), e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131:33199)
I20240501 14:16:01.846180 14418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca266243bcb84e8c9b65d1b096b5725d" candidate_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17062e38120042aaaca05e153b2c6ccf"
I20240501 14:16:01.846197 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca266243bcb84e8c9b65d1b096b5725d" candidate_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e625ab1b03d14e2aae194052d2c67bc6"
I20240501 14:16:01.846294 14418 raft_consensus.cc:3037] T ca266243bcb84e8c9b65d1b096b5725d P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:01.846295 14549 raft_consensus.cc:3037] T ca266243bcb84e8c9b65d1b096b5725d P e625ab1b03d14e2aae194052d2c67bc6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:01.847110 14418 raft_consensus.cc:2454] T ca266243bcb84e8c9b65d1b096b5725d P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c87cbbf2d3ab4143815ef2c74ae760db in term 1.
I20240501 14:16:01.847127 14549 raft_consensus.cc:2454] T ca266243bcb84e8c9b65d1b096b5725d P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c87cbbf2d3ab4143815ef2c74ae760db in term 1.
I20240501 14:16:01.847359 14220 leader_election.cc:302] T ca266243bcb84e8c9b65d1b096b5725d P c87cbbf2d3ab4143815ef2c74ae760db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17062e38120042aaaca05e153b2c6ccf, c87cbbf2d3ab4143815ef2c74ae760db; no voters:
I20240501 14:16:01.847494 14951 raft_consensus.cc:2781] T ca266243bcb84e8c9b65d1b096b5725d P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:16:01.847581 14951 raft_consensus.cc:686] T ca266243bcb84e8c9b65d1b096b5725d P c87cbbf2d3ab4143815ef2c74ae760db [term 1 LEADER]: Becoming Leader. State: Replica: c87cbbf2d3ab4143815ef2c74ae760db, State: Running, Role: LEADER
I20240501 14:16:01.847712 14951 consensus_queue.cc:237] T ca266243bcb84e8c9b65d1b096b5725d P c87cbbf2d3ab4143815ef2c74ae760db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:01.849129 14141 catalog_manager.cc:5484] T ca266243bcb84e8c9b65d1b096b5725d P c87cbbf2d3ab4143815ef2c74ae760db reported cstate change: term changed from 0 to 1, leader changed from <none> to c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129). New cstate: current_term: 1 leader_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } health_report { overall_health: UNKNOWN } } }
I20240501 14:16:01.889140 14901 raft_consensus.cc:482] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:16:01.889279 14901 raft_consensus.cc:504] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:01.889725 14901 leader_election.cc:288] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129:42369), e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131:33199)
I20240501 14:16:01.890123 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40732c47173c429aa69c6a41475849de" candidate_uuid: "17062e38120042aaaca05e153b2c6ccf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e625ab1b03d14e2aae194052d2c67bc6" is_pre_election: true
I20240501 14:16:01.890172 14286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40732c47173c429aa69c6a41475849de" candidate_uuid: "17062e38120042aaaca05e153b2c6ccf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" is_pre_election: true
I20240501 14:16:01.890271 14549 raft_consensus.cc:2454] T 40732c47173c429aa69c6a41475849de P e625ab1b03d14e2aae194052d2c67bc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17062e38120042aaaca05e153b2c6ccf in term 0.
I20240501 14:16:01.890321 14286 raft_consensus.cc:2454] T 40732c47173c429aa69c6a41475849de P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17062e38120042aaaca05e153b2c6ccf in term 0.
I20240501 14:16:01.890605 14354 leader_election.cc:302] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17062e38120042aaaca05e153b2c6ccf, e625ab1b03d14e2aae194052d2c67bc6; no voters:
I20240501 14:16:01.890781 14901 raft_consensus.cc:2781] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:16:01.890843 14901 raft_consensus.cc:482] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:16:01.890889 14901 raft_consensus.cc:3037] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:01.891804 14901 raft_consensus.cc:504] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:01.892136 14901 leader_election.cc:288] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf [CANDIDATE]: Term 1 election: Requested vote from peers c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129:42369), e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131:33199)
I20240501 14:16:01.892480 14286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40732c47173c429aa69c6a41475849de" candidate_uuid: "17062e38120042aaaca05e153b2c6ccf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c87cbbf2d3ab4143815ef2c74ae760db"
I20240501 14:16:01.892504 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40732c47173c429aa69c6a41475849de" candidate_uuid: "17062e38120042aaaca05e153b2c6ccf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e625ab1b03d14e2aae194052d2c67bc6"
I20240501 14:16:01.892593 14286 raft_consensus.cc:3037] T 40732c47173c429aa69c6a41475849de P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:01.892608 14549 raft_consensus.cc:3037] T 40732c47173c429aa69c6a41475849de P e625ab1b03d14e2aae194052d2c67bc6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:01.893353 14286 raft_consensus.cc:2454] T 40732c47173c429aa69c6a41475849de P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17062e38120042aaaca05e153b2c6ccf in term 1.
I20240501 14:16:01.893360 14549 raft_consensus.cc:2454] T 40732c47173c429aa69c6a41475849de P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17062e38120042aaaca05e153b2c6ccf in term 1.
I20240501 14:16:01.893647 14354 leader_election.cc:302] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 17062e38120042aaaca05e153b2c6ccf, c87cbbf2d3ab4143815ef2c74ae760db, e625ab1b03d14e2aae194052d2c67bc6; no voters:
I20240501 14:16:01.893855 14901 raft_consensus.cc:2781] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:16:01.893950 14901 raft_consensus.cc:686] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf [term 1 LEADER]: Becoming Leader. State: Replica: 17062e38120042aaaca05e153b2c6ccf, State: Running, Role: LEADER
I20240501 14:16:01.894099 14901 consensus_queue.cc:237] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:01.895658 14141 catalog_manager.cc:5484] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf reported cstate change: term changed from 0 to 1, leader changed from <none> to 17062e38120042aaaca05e153b2c6ccf (127.7.39.130). New cstate: current_term: 1 leader_uuid: "17062e38120042aaaca05e153b2c6ccf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } health_report { overall_health: UNKNOWN } } }
I20240501 14:16:01.921581 14885 raft_consensus.cc:482] T 8ebd7203c0914f6c8e26ab4d20074765 P e625ab1b03d14e2aae194052d2c67bc6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:16:01.921722 14885 raft_consensus.cc:504] T 8ebd7203c0914f6c8e26ab4d20074765 P e625ab1b03d14e2aae194052d2c67bc6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:01.922924 14885 leader_election.cc:288] T 8ebd7203c0914f6c8e26ab4d20074765 P e625ab1b03d14e2aae194052d2c67bc6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129:42369), 17062e38120042aaaca05e153b2c6ccf (127.7.39.130:43603)
I20240501 14:16:01.927367 14286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ebd7203c0914f6c8e26ab4d20074765" candidate_uuid: "e625ab1b03d14e2aae194052d2c67bc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" is_pre_election: true
I20240501 14:16:01.927381 14418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ebd7203c0914f6c8e26ab4d20074765" candidate_uuid: "e625ab1b03d14e2aae194052d2c67bc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17062e38120042aaaca05e153b2c6ccf" is_pre_election: true
I20240501 14:16:01.927515 14286 raft_consensus.cc:2454] T 8ebd7203c0914f6c8e26ab4d20074765 P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e625ab1b03d14e2aae194052d2c67bc6 in term 0.
I20240501 14:16:01.927520 14418 raft_consensus.cc:2454] T 8ebd7203c0914f6c8e26ab4d20074765 P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e625ab1b03d14e2aae194052d2c67bc6 in term 0.
I20240501 14:16:01.927851 14484 leader_election.cc:302] T 8ebd7203c0914f6c8e26ab4d20074765 P e625ab1b03d14e2aae194052d2c67bc6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17062e38120042aaaca05e153b2c6ccf, e625ab1b03d14e2aae194052d2c67bc6; no voters:
I20240501 14:16:01.928088 14885 raft_consensus.cc:2781] T 8ebd7203c0914f6c8e26ab4d20074765 P e625ab1b03d14e2aae194052d2c67bc6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:16:01.928151 14885 raft_consensus.cc:482] T 8ebd7203c0914f6c8e26ab4d20074765 P e625ab1b03d14e2aae194052d2c67bc6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:16:01.928196 14885 raft_consensus.cc:3037] T 8ebd7203c0914f6c8e26ab4d20074765 P e625ab1b03d14e2aae194052d2c67bc6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:01.929006 14885 raft_consensus.cc:504] T 8ebd7203c0914f6c8e26ab4d20074765 P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:01.929369 14885 leader_election.cc:288] T 8ebd7203c0914f6c8e26ab4d20074765 P e625ab1b03d14e2aae194052d2c67bc6 [CANDIDATE]: Term 1 election: Requested vote from peers c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129:42369), 17062e38120042aaaca05e153b2c6ccf (127.7.39.130:43603)
I20240501 14:16:01.929713 14286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ebd7203c0914f6c8e26ab4d20074765" candidate_uuid: "e625ab1b03d14e2aae194052d2c67bc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c87cbbf2d3ab4143815ef2c74ae760db"
I20240501 14:16:01.929723 14418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ebd7203c0914f6c8e26ab4d20074765" candidate_uuid: "e625ab1b03d14e2aae194052d2c67bc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17062e38120042aaaca05e153b2c6ccf"
I20240501 14:16:01.929833 14418 raft_consensus.cc:3037] T 8ebd7203c0914f6c8e26ab4d20074765 P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:01.929833 14286 raft_consensus.cc:3037] T 8ebd7203c0914f6c8e26ab4d20074765 P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:01.930869 14418 raft_consensus.cc:2454] T 8ebd7203c0914f6c8e26ab4d20074765 P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e625ab1b03d14e2aae194052d2c67bc6 in term 1.
I20240501 14:16:01.931151 14484 leader_election.cc:302] T 8ebd7203c0914f6c8e26ab4d20074765 P e625ab1b03d14e2aae194052d2c67bc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17062e38120042aaaca05e153b2c6ccf, e625ab1b03d14e2aae194052d2c67bc6; no voters:
I20240501 14:16:01.931326 14286 raft_consensus.cc:2454] T 8ebd7203c0914f6c8e26ab4d20074765 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e625ab1b03d14e2aae194052d2c67bc6 in term 1.
I20240501 14:16:01.931370 14885 raft_consensus.cc:2781] T 8ebd7203c0914f6c8e26ab4d20074765 P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:16:01.931594 14885 raft_consensus.cc:686] T 8ebd7203c0914f6c8e26ab4d20074765 P e625ab1b03d14e2aae194052d2c67bc6 [term 1 LEADER]: Becoming Leader. State: Replica: e625ab1b03d14e2aae194052d2c67bc6, State: Running, Role: LEADER
I20240501 14:16:01.931939 14885 consensus_queue.cc:237] T 8ebd7203c0914f6c8e26ab4d20074765 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:01.934130 14141 catalog_manager.cc:5484] T 8ebd7203c0914f6c8e26ab4d20074765 P e625ab1b03d14e2aae194052d2c67bc6 reported cstate change: term changed from 0 to 1, leader changed from <none> to e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131). New cstate: current_term: 1 leader_uuid: "e625ab1b03d14e2aae194052d2c67bc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } health_report { overall_health: HEALTHY } } }
W20240501 14:16:02.073379 14977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:16:02.073621 14977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:16:02.076721 14977 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 14:16:02.172806 14286 raft_consensus.cc:1261] T 40732c47173c429aa69c6a41475849de P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Refusing update from remote peer 17062e38120042aaaca05e153b2c6ccf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:16:02.173138 14549 raft_consensus.cc:1261] T ca266243bcb84e8c9b65d1b096b5725d P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Refusing update from remote peer c87cbbf2d3ab4143815ef2c74ae760db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:16:02.173161 14418 raft_consensus.cc:1261] T ca266243bcb84e8c9b65d1b096b5725d P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Refusing update from remote peer c87cbbf2d3ab4143815ef2c74ae760db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:16:02.173312 14892 consensus_queue.cc:1035] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:16:02.173673 14951 consensus_queue.cc:1035] T ca266243bcb84e8c9b65d1b096b5725d P c87cbbf2d3ab4143815ef2c74ae760db [LEADER]: Connected to new peer: Peer: permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:16:02.173787 14883 consensus_queue.cc:1035] T ca266243bcb84e8c9b65d1b096b5725d P c87cbbf2d3ab4143815ef2c74ae760db [LEADER]: Connected to new peer: Peer: permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:16:02.173904 14549 raft_consensus.cc:1261] T 40732c47173c429aa69c6a41475849de P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Refusing update from remote peer 17062e38120042aaaca05e153b2c6ccf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:16:02.174460 14892 consensus_queue.cc:1035] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:16:02.174561 14418 raft_consensus.cc:1261] T 8ebd7203c0914f6c8e26ab4d20074765 P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Refusing update from remote peer e625ab1b03d14e2aae194052d2c67bc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:16:02.179229 14285 raft_consensus.cc:1261] T 8ebd7203c0914f6c8e26ab4d20074765 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Refusing update from remote peer e625ab1b03d14e2aae194052d2c67bc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:16:02.179824 14975 consensus_queue.cc:1035] T 8ebd7203c0914f6c8e26ab4d20074765 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:16:02.180217 14975 consensus_queue.cc:1035] T 8ebd7203c0914f6c8e26ab4d20074765 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 120.378 ms
time per row: 0.060189 ms
I20240501 14:16:02.239590 14142 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240501 14:16:02.240124 14142 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 14:16:02.245339 14266 tablet_service.cc:1467] Processing CreateTablet for tablet acd35bd40253406fa1baa3f0d19c3d0a (DEFAULT_TABLE table=rebalance_test_table_2 [id=1176c470113c40c796085916c4a929d7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:02.245524 14397 tablet_service.cc:1467] Processing CreateTablet for tablet acd35bd40253406fa1baa3f0d19c3d0a (DEFAULT_TABLE table=rebalance_test_table_2 [id=1176c470113c40c796085916c4a929d7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:02.245679 14266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acd35bd40253406fa1baa3f0d19c3d0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:02.245918 14397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acd35bd40253406fa1baa3f0d19c3d0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:02.247526 14961 tablet_bootstrap.cc:492] T acd35bd40253406fa1baa3f0d19c3d0a P c87cbbf2d3ab4143815ef2c74ae760db: Bootstrap starting.
I20240501 14:16:02.247905 14884 tablet_bootstrap.cc:492] T acd35bd40253406fa1baa3f0d19c3d0a P 17062e38120042aaaca05e153b2c6ccf: Bootstrap starting.
I20240501 14:16:02.248102 14266 tablet_service.cc:1467] Processing CreateTablet for tablet 47ef7ee4ea3b46cbbac2126ceaa5fa24 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1176c470113c40c796085916c4a929d7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:02.248153 14264 tablet_service.cc:1467] Processing CreateTablet for tablet 00618ea639cf49d09cc3523b47cd0c07 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1176c470113c40c796085916c4a929d7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:02.248314 14266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47ef7ee4ea3b46cbbac2126ceaa5fa24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:02.248389 14961 tablet_bootstrap.cc:654] T acd35bd40253406fa1baa3f0d19c3d0a P c87cbbf2d3ab4143815ef2c74ae760db: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:02.248485 14264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00618ea639cf49d09cc3523b47cd0c07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:02.248765 14884 tablet_bootstrap.cc:654] T acd35bd40253406fa1baa3f0d19c3d0a P 17062e38120042aaaca05e153b2c6ccf: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:02.249557 14530 tablet_service.cc:1467] Processing CreateTablet for tablet acd35bd40253406fa1baa3f0d19c3d0a (DEFAULT_TABLE table=rebalance_test_table_2 [id=1176c470113c40c796085916c4a929d7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:02.249827 14530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acd35bd40253406fa1baa3f0d19c3d0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:02.250106 14398 tablet_service.cc:1467] Processing CreateTablet for tablet 00618ea639cf49d09cc3523b47cd0c07 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1176c470113c40c796085916c4a929d7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:02.250334 14398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00618ea639cf49d09cc3523b47cd0c07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:02.250942 14961 tablet_bootstrap.cc:492] T acd35bd40253406fa1baa3f0d19c3d0a P c87cbbf2d3ab4143815ef2c74ae760db: No bootstrap required, opened a new log
I20240501 14:16:02.251020 14961 ts_tablet_manager.cc:1367] T acd35bd40253406fa1baa3f0d19c3d0a P c87cbbf2d3ab4143815ef2c74ae760db: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:16:02.251534 14961 raft_consensus.cc:348] T acd35bd40253406fa1baa3f0d19c3d0a P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:02.251654 14961 raft_consensus.cc:374] T acd35bd40253406fa1baa3f0d19c3d0a P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:02.251688 14961 raft_consensus.cc:729] T acd35bd40253406fa1baa3f0d19c3d0a P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c87cbbf2d3ab4143815ef2c74ae760db, State: Initialized, Role: FOLLOWER
I20240501 14:16:02.251809 14961 consensus_queue.cc:260] T acd35bd40253406fa1baa3f0d19c3d0a P c87cbbf2d3ab4143815ef2c74ae760db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:02.251875 14960 tablet_bootstrap.cc:492] T acd35bd40253406fa1baa3f0d19c3d0a P e625ab1b03d14e2aae194052d2c67bc6: Bootstrap starting.
I20240501 14:16:02.252061 14529 tablet_service.cc:1467] Processing CreateTablet for tablet 47ef7ee4ea3b46cbbac2126ceaa5fa24 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1176c470113c40c796085916c4a929d7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:02.252128 14961 ts_tablet_manager.cc:1398] T acd35bd40253406fa1baa3f0d19c3d0a P c87cbbf2d3ab4143815ef2c74ae760db: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:16:02.252290 14529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47ef7ee4ea3b46cbbac2126ceaa5fa24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:02.252300 14961 tablet_bootstrap.cc:492] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P c87cbbf2d3ab4143815ef2c74ae760db: Bootstrap starting.
I20240501 14:16:02.252816 14960 tablet_bootstrap.cc:654] T acd35bd40253406fa1baa3f0d19c3d0a P e625ab1b03d14e2aae194052d2c67bc6: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:02.253249 14961 tablet_bootstrap.cc:654] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P c87cbbf2d3ab4143815ef2c74ae760db: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:02.254288 14528 tablet_service.cc:1467] Processing CreateTablet for tablet 00618ea639cf49d09cc3523b47cd0c07 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1176c470113c40c796085916c4a929d7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:02.254521 14528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00618ea639cf49d09cc3523b47cd0c07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:02.254623 14961 tablet_bootstrap.cc:492] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P c87cbbf2d3ab4143815ef2c74ae760db: No bootstrap required, opened a new log
I20240501 14:16:02.254695 14961 ts_tablet_manager.cc:1367] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P c87cbbf2d3ab4143815ef2c74ae760db: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:16:02.254724 14397 tablet_service.cc:1467] Processing CreateTablet for tablet 47ef7ee4ea3b46cbbac2126ceaa5fa24 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1176c470113c40c796085916c4a929d7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:02.254943 14397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47ef7ee4ea3b46cbbac2126ceaa5fa24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:02.255192 14961 raft_consensus.cc:348] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:02.255309 14961 raft_consensus.cc:374] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:02.255411 14961 raft_consensus.cc:729] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c87cbbf2d3ab4143815ef2c74ae760db, State: Initialized, Role: FOLLOWER
I20240501 14:16:02.255542 14961 consensus_queue.cc:260] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P c87cbbf2d3ab4143815ef2c74ae760db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:02.255862 14961 ts_tablet_manager.cc:1398] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P c87cbbf2d3ab4143815ef2c74ae760db: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:16:02.256083 14961 tablet_bootstrap.cc:492] T 00618ea639cf49d09cc3523b47cd0c07 P c87cbbf2d3ab4143815ef2c74ae760db: Bootstrap starting.
I20240501 14:16:02.257079 14961 tablet_bootstrap.cc:654] T 00618ea639cf49d09cc3523b47cd0c07 P c87cbbf2d3ab4143815ef2c74ae760db: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:02.257694 14884 tablet_bootstrap.cc:492] T acd35bd40253406fa1baa3f0d19c3d0a P 17062e38120042aaaca05e153b2c6ccf: No bootstrap required, opened a new log
I20240501 14:16:02.257792 14884 ts_tablet_manager.cc:1367] T acd35bd40253406fa1baa3f0d19c3d0a P 17062e38120042aaaca05e153b2c6ccf: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240501 14:16:02.257892 14960 tablet_bootstrap.cc:492] T acd35bd40253406fa1baa3f0d19c3d0a P e625ab1b03d14e2aae194052d2c67bc6: No bootstrap required, opened a new log
I20240501 14:16:02.257983 14960 ts_tablet_manager.cc:1367] T acd35bd40253406fa1baa3f0d19c3d0a P e625ab1b03d14e2aae194052d2c67bc6: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:16:02.258366 14884 raft_consensus.cc:348] T acd35bd40253406fa1baa3f0d19c3d0a P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:02.258481 14884 raft_consensus.cc:374] T acd35bd40253406fa1baa3f0d19c3d0a P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:02.258515 14884 raft_consensus.cc:729] T acd35bd40253406fa1baa3f0d19c3d0a P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17062e38120042aaaca05e153b2c6ccf, State: Initialized, Role: FOLLOWER
I20240501 14:16:02.258558 14960 raft_consensus.cc:348] T acd35bd40253406fa1baa3f0d19c3d0a P e625ab1b03d14e2aae194052d2c67bc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:02.258664 14960 raft_consensus.cc:374] T acd35bd40253406fa1baa3f0d19c3d0a P e625ab1b03d14e2aae194052d2c67bc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:02.258697 14960 raft_consensus.cc:729] T acd35bd40253406fa1baa3f0d19c3d0a P e625ab1b03d14e2aae194052d2c67bc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e625ab1b03d14e2aae194052d2c67bc6, State: Initialized, Role: FOLLOWER
I20240501 14:16:02.258647 14884 consensus_queue.cc:260] T acd35bd40253406fa1baa3f0d19c3d0a P 17062e38120042aaaca05e153b2c6ccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:02.258800 14960 consensus_queue.cc:260] T acd35bd40253406fa1baa3f0d19c3d0a P e625ab1b03d14e2aae194052d2c67bc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:02.258946 14884 ts_tablet_manager.cc:1398] T acd35bd40253406fa1baa3f0d19c3d0a P 17062e38120042aaaca05e153b2c6ccf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:16:02.259096 14960 ts_tablet_manager.cc:1398] T acd35bd40253406fa1baa3f0d19c3d0a P e625ab1b03d14e2aae194052d2c67bc6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:16:02.259253 14960 tablet_bootstrap.cc:492] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6: Bootstrap starting.
I20240501 14:16:02.259873 14884 tablet_bootstrap.cc:492] T 00618ea639cf49d09cc3523b47cd0c07 P 17062e38120042aaaca05e153b2c6ccf: Bootstrap starting.
I20240501 14:16:02.260107 14961 tablet_bootstrap.cc:492] T 00618ea639cf49d09cc3523b47cd0c07 P c87cbbf2d3ab4143815ef2c74ae760db: No bootstrap required, opened a new log
I20240501 14:16:02.260135 14960 tablet_bootstrap.cc:654] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:02.260183 14961 ts_tablet_manager.cc:1367] T 00618ea639cf49d09cc3523b47cd0c07 P c87cbbf2d3ab4143815ef2c74ae760db: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:16:02.260627 14961 raft_consensus.cc:348] T 00618ea639cf49d09cc3523b47cd0c07 P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:02.260721 14961 raft_consensus.cc:374] T 00618ea639cf49d09cc3523b47cd0c07 P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:02.260752 14961 raft_consensus.cc:729] T 00618ea639cf49d09cc3523b47cd0c07 P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c87cbbf2d3ab4143815ef2c74ae760db, State: Initialized, Role: FOLLOWER
I20240501 14:16:02.260885 14961 consensus_queue.cc:260] T 00618ea639cf49d09cc3523b47cd0c07 P c87cbbf2d3ab4143815ef2c74ae760db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:02.261185 14961 ts_tablet_manager.cc:1398] T 00618ea639cf49d09cc3523b47cd0c07 P c87cbbf2d3ab4143815ef2c74ae760db: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:16:02.261821 14884 tablet_bootstrap.cc:654] T 00618ea639cf49d09cc3523b47cd0c07 P 17062e38120042aaaca05e153b2c6ccf: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:02.262766 14960 tablet_bootstrap.cc:492] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6: No bootstrap required, opened a new log
I20240501 14:16:02.262863 14960 ts_tablet_manager.cc:1367] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:16:02.262890 14884 tablet_bootstrap.cc:492] T 00618ea639cf49d09cc3523b47cd0c07 P 17062e38120042aaaca05e153b2c6ccf: No bootstrap required, opened a new log
I20240501 14:16:02.262979 14884 ts_tablet_manager.cc:1367] T 00618ea639cf49d09cc3523b47cd0c07 P 17062e38120042aaaca05e153b2c6ccf: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:16:02.263360 14960 raft_consensus.cc:348] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:02.263463 14960 raft_consensus.cc:374] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:02.263502 14960 raft_consensus.cc:729] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e625ab1b03d14e2aae194052d2c67bc6, State: Initialized, Role: FOLLOWER
I20240501 14:16:02.263470 14884 raft_consensus.cc:348] T 00618ea639cf49d09cc3523b47cd0c07 P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:02.263571 14884 raft_consensus.cc:374] T 00618ea639cf49d09cc3523b47cd0c07 P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:02.263614 14884 raft_consensus.cc:729] T 00618ea639cf49d09cc3523b47cd0c07 P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17062e38120042aaaca05e153b2c6ccf, State: Initialized, Role: FOLLOWER
I20240501 14:16:02.263649 14960 consensus_queue.cc:260] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:02.263727 14884 consensus_queue.cc:260] T 00618ea639cf49d09cc3523b47cd0c07 P 17062e38120042aaaca05e153b2c6ccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:02.263999 14960 ts_tablet_manager.cc:1398] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:16:02.264119 14884 ts_tablet_manager.cc:1398] T 00618ea639cf49d09cc3523b47cd0c07 P 17062e38120042aaaca05e153b2c6ccf: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:16:02.264191 14960 tablet_bootstrap.cc:492] T 00618ea639cf49d09cc3523b47cd0c07 P e625ab1b03d14e2aae194052d2c67bc6: Bootstrap starting.
I20240501 14:16:02.264288 14884 tablet_bootstrap.cc:492] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 17062e38120042aaaca05e153b2c6ccf: Bootstrap starting.
I20240501 14:16:02.265014 14960 tablet_bootstrap.cc:654] T 00618ea639cf49d09cc3523b47cd0c07 P e625ab1b03d14e2aae194052d2c67bc6: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:02.265477 14884 tablet_bootstrap.cc:654] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 17062e38120042aaaca05e153b2c6ccf: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:02.266161 14960 tablet_bootstrap.cc:492] T 00618ea639cf49d09cc3523b47cd0c07 P e625ab1b03d14e2aae194052d2c67bc6: No bootstrap required, opened a new log
I20240501 14:16:02.266263 14960 ts_tablet_manager.cc:1367] T 00618ea639cf49d09cc3523b47cd0c07 P e625ab1b03d14e2aae194052d2c67bc6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:16:02.266656 14960 raft_consensus.cc:348] T 00618ea639cf49d09cc3523b47cd0c07 P e625ab1b03d14e2aae194052d2c67bc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:02.266743 14960 raft_consensus.cc:374] T 00618ea639cf49d09cc3523b47cd0c07 P e625ab1b03d14e2aae194052d2c67bc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:02.266780 14960 raft_consensus.cc:729] T 00618ea639cf49d09cc3523b47cd0c07 P e625ab1b03d14e2aae194052d2c67bc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e625ab1b03d14e2aae194052d2c67bc6, State: Initialized, Role: FOLLOWER
I20240501 14:16:02.266904 14960 consensus_queue.cc:260] T 00618ea639cf49d09cc3523b47cd0c07 P e625ab1b03d14e2aae194052d2c67bc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:02.267191 14960 ts_tablet_manager.cc:1398] T 00618ea639cf49d09cc3523b47cd0c07 P e625ab1b03d14e2aae194052d2c67bc6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:16:02.267812 14884 tablet_bootstrap.cc:492] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 17062e38120042aaaca05e153b2c6ccf: No bootstrap required, opened a new log
I20240501 14:16:02.267916 14884 ts_tablet_manager.cc:1367] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 17062e38120042aaaca05e153b2c6ccf: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:16:02.268376 14884 raft_consensus.cc:348] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:02.268489 14884 raft_consensus.cc:374] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:02.268536 14884 raft_consensus.cc:729] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17062e38120042aaaca05e153b2c6ccf, State: Initialized, Role: FOLLOWER
I20240501 14:16:02.268659 14884 consensus_queue.cc:260] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 17062e38120042aaaca05e153b2c6ccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:02.268900 14884 ts_tablet_manager.cc:1398] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 17062e38120042aaaca05e153b2c6ccf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:16:02.286339 14892 raft_consensus.cc:482] T 00618ea639cf49d09cc3523b47cd0c07 P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:16:02.286433 14892 raft_consensus.cc:504] T 00618ea639cf49d09cc3523b47cd0c07 P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:02.286780 14892 leader_election.cc:288] T 00618ea639cf49d09cc3523b47cd0c07 P 17062e38120042aaaca05e153b2c6ccf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129:42369), e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131:33199)
I20240501 14:16:02.287155 14285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00618ea639cf49d09cc3523b47cd0c07" candidate_uuid: "17062e38120042aaaca05e153b2c6ccf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" is_pre_election: true
I20240501 14:16:02.287192 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00618ea639cf49d09cc3523b47cd0c07" candidate_uuid: "17062e38120042aaaca05e153b2c6ccf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e625ab1b03d14e2aae194052d2c67bc6" is_pre_election: true
I20240501 14:16:02.287290 14285 raft_consensus.cc:2454] T 00618ea639cf49d09cc3523b47cd0c07 P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17062e38120042aaaca05e153b2c6ccf in term 0.
I20240501 14:16:02.287334 14548 raft_consensus.cc:2454] T 00618ea639cf49d09cc3523b47cd0c07 P e625ab1b03d14e2aae194052d2c67bc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17062e38120042aaaca05e153b2c6ccf in term 0.
I20240501 14:16:02.287578 14351 leader_election.cc:302] T 00618ea639cf49d09cc3523b47cd0c07 P 17062e38120042aaaca05e153b2c6ccf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17062e38120042aaaca05e153b2c6ccf, c87cbbf2d3ab4143815ef2c74ae760db; no voters:
I20240501 14:16:02.287740 14892 raft_consensus.cc:2781] T 00618ea639cf49d09cc3523b47cd0c07 P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:16:02.287801 14892 raft_consensus.cc:482] T 00618ea639cf49d09cc3523b47cd0c07 P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:16:02.287825 14892 raft_consensus.cc:3037] T 00618ea639cf49d09cc3523b47cd0c07 P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:02.288687 14892 raft_consensus.cc:504] T 00618ea639cf49d09cc3523b47cd0c07 P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:02.288981 14892 leader_election.cc:288] T 00618ea639cf49d09cc3523b47cd0c07 P 17062e38120042aaaca05e153b2c6ccf [CANDIDATE]: Term 1 election: Requested vote from peers c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129:42369), e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131:33199)
I20240501 14:16:02.289258 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00618ea639cf49d09cc3523b47cd0c07" candidate_uuid: "17062e38120042aaaca05e153b2c6ccf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e625ab1b03d14e2aae194052d2c67bc6"
I20240501 14:16:02.289268 14285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00618ea639cf49d09cc3523b47cd0c07" candidate_uuid: "17062e38120042aaaca05e153b2c6ccf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c87cbbf2d3ab4143815ef2c74ae760db"
I20240501 14:16:02.289346 14285 raft_consensus.cc:3037] T 00618ea639cf49d09cc3523b47cd0c07 P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:02.289355 14548 raft_consensus.cc:3037] T 00618ea639cf49d09cc3523b47cd0c07 P e625ab1b03d14e2aae194052d2c67bc6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:02.290067 14285 raft_consensus.cc:2454] T 00618ea639cf49d09cc3523b47cd0c07 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17062e38120042aaaca05e153b2c6ccf in term 1.
I20240501 14:16:02.290136 14548 raft_consensus.cc:2454] T 00618ea639cf49d09cc3523b47cd0c07 P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17062e38120042aaaca05e153b2c6ccf in term 1.
I20240501 14:16:02.290336 14351 leader_election.cc:302] T 00618ea639cf49d09cc3523b47cd0c07 P 17062e38120042aaaca05e153b2c6ccf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17062e38120042aaaca05e153b2c6ccf, c87cbbf2d3ab4143815ef2c74ae760db; no voters:
I20240501 14:16:02.290530 14892 raft_consensus.cc:2781] T 00618ea639cf49d09cc3523b47cd0c07 P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:16:02.290598 14892 raft_consensus.cc:686] T 00618ea639cf49d09cc3523b47cd0c07 P 17062e38120042aaaca05e153b2c6ccf [term 1 LEADER]: Becoming Leader. State: Replica: 17062e38120042aaaca05e153b2c6ccf, State: Running, Role: LEADER
I20240501 14:16:02.290729 14892 consensus_queue.cc:237] T 00618ea639cf49d09cc3523b47cd0c07 P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:02.292151 14136 catalog_manager.cc:5484] T 00618ea639cf49d09cc3523b47cd0c07 P 17062e38120042aaaca05e153b2c6ccf reported cstate change: term changed from 0 to 1, leader changed from <none> to 17062e38120042aaaca05e153b2c6ccf (127.7.39.130). New cstate: current_term: 1 leader_uuid: "17062e38120042aaaca05e153b2c6ccf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } health_report { overall_health: UNKNOWN } } }
I20240501 14:16:02.472790 14975 raft_consensus.cc:482] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:16:02.472934 14975 raft_consensus.cc:504] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:02.473598 14975 leader_election.cc:288] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129:42369), 17062e38120042aaaca05e153b2c6ccf (127.7.39.130:43603)
I20240501 14:16:02.473955 14285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ef7ee4ea3b46cbbac2126ceaa5fa24" candidate_uuid: "e625ab1b03d14e2aae194052d2c67bc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" is_pre_election: true
I20240501 14:16:02.474102 14285 raft_consensus.cc:2454] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e625ab1b03d14e2aae194052d2c67bc6 in term 0.
I20240501 14:16:02.474056 14417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ef7ee4ea3b46cbbac2126ceaa5fa24" candidate_uuid: "e625ab1b03d14e2aae194052d2c67bc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17062e38120042aaaca05e153b2c6ccf" is_pre_election: true
I20240501 14:16:02.474226 14417 raft_consensus.cc:2454] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e625ab1b03d14e2aae194052d2c67bc6 in term 0.
I20240501 14:16:02.474488 14483 leader_election.cc:302] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c87cbbf2d3ab4143815ef2c74ae760db, e625ab1b03d14e2aae194052d2c67bc6; no voters:
I20240501 14:16:02.474658 14975 raft_consensus.cc:2781] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:16:02.474721 14975 raft_consensus.cc:482] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:16:02.474776 14975 raft_consensus.cc:3037] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:02.475710 14975 raft_consensus.cc:504] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:02.476070 14975 leader_election.cc:288] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6 [CANDIDATE]: Term 1 election: Requested vote from peers c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129:42369), 17062e38120042aaaca05e153b2c6ccf (127.7.39.130:43603)
I20240501 14:16:02.476388 14285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ef7ee4ea3b46cbbac2126ceaa5fa24" candidate_uuid: "e625ab1b03d14e2aae194052d2c67bc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c87cbbf2d3ab4143815ef2c74ae760db"
I20240501 14:16:02.476435 14417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ef7ee4ea3b46cbbac2126ceaa5fa24" candidate_uuid: "e625ab1b03d14e2aae194052d2c67bc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17062e38120042aaaca05e153b2c6ccf"
I20240501 14:16:02.476542 14417 raft_consensus.cc:3037] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:02.476543 14285 raft_consensus.cc:3037] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:02.477473 14285 raft_consensus.cc:2454] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e625ab1b03d14e2aae194052d2c67bc6 in term 1.
I20240501 14:16:02.477473 14417 raft_consensus.cc:2454] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e625ab1b03d14e2aae194052d2c67bc6 in term 1.
I20240501 14:16:02.477766 14483 leader_election.cc:302] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c87cbbf2d3ab4143815ef2c74ae760db, e625ab1b03d14e2aae194052d2c67bc6; no voters:
I20240501 14:16:02.477998 14975 raft_consensus.cc:2781] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:16:02.478116 14975 raft_consensus.cc:686] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6 [term 1 LEADER]: Becoming Leader. State: Replica: e625ab1b03d14e2aae194052d2c67bc6, State: Running, Role: LEADER
I20240501 14:16:02.478266 14975 consensus_queue.cc:237] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:02.480018 14136 catalog_manager.cc:5484] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6 reported cstate change: term changed from 0 to 1, leader changed from <none> to e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131). New cstate: current_term: 1 leader_uuid: "e625ab1b03d14e2aae194052d2c67bc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } health_report { overall_health: HEALTHY } } }
I20240501 14:16:02.488559 14897 raft_consensus.cc:482] T acd35bd40253406fa1baa3f0d19c3d0a P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:16:02.488662 14897 raft_consensus.cc:504] T acd35bd40253406fa1baa3f0d19c3d0a P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:02.489024 14897 leader_election.cc:288] T acd35bd40253406fa1baa3f0d19c3d0a P c87cbbf2d3ab4143815ef2c74ae760db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17062e38120042aaaca05e153b2c6ccf (127.7.39.130:43603), e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131:33199)
I20240501 14:16:02.489362 14417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acd35bd40253406fa1baa3f0d19c3d0a" candidate_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17062e38120042aaaca05e153b2c6ccf" is_pre_election: true
I20240501 14:16:02.489388 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acd35bd40253406fa1baa3f0d19c3d0a" candidate_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e625ab1b03d14e2aae194052d2c67bc6" is_pre_election: true
I20240501 14:16:02.489471 14417 raft_consensus.cc:2454] T acd35bd40253406fa1baa3f0d19c3d0a P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c87cbbf2d3ab4143815ef2c74ae760db in term 0.
I20240501 14:16:02.489506 14548 raft_consensus.cc:2454] T acd35bd40253406fa1baa3f0d19c3d0a P e625ab1b03d14e2aae194052d2c67bc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c87cbbf2d3ab4143815ef2c74ae760db in term 0.
I20240501 14:16:02.489758 14220 leader_election.cc:302] T acd35bd40253406fa1baa3f0d19c3d0a P c87cbbf2d3ab4143815ef2c74ae760db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17062e38120042aaaca05e153b2c6ccf, c87cbbf2d3ab4143815ef2c74ae760db; no voters:
I20240501 14:16:02.489945 14897 raft_consensus.cc:2781] T acd35bd40253406fa1baa3f0d19c3d0a P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:16:02.490003 14897 raft_consensus.cc:482] T acd35bd40253406fa1baa3f0d19c3d0a P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:16:02.490039 14897 raft_consensus.cc:3037] T acd35bd40253406fa1baa3f0d19c3d0a P c87cbbf2d3ab4143815ef2c74ae760db [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:02.490752 14897 raft_consensus.cc:504] T acd35bd40253406fa1baa3f0d19c3d0a P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:02.491006 14897 leader_election.cc:288] T acd35bd40253406fa1baa3f0d19c3d0a P c87cbbf2d3ab4143815ef2c74ae760db [CANDIDATE]: Term 1 election: Requested vote from peers 17062e38120042aaaca05e153b2c6ccf (127.7.39.130:43603), e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131:33199)
I20240501 14:16:02.491317 14417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acd35bd40253406fa1baa3f0d19c3d0a" candidate_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17062e38120042aaaca05e153b2c6ccf"
I20240501 14:16:02.491334 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acd35bd40253406fa1baa3f0d19c3d0a" candidate_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e625ab1b03d14e2aae194052d2c67bc6"
I20240501 14:16:02.491415 14417 raft_consensus.cc:3037] T acd35bd40253406fa1baa3f0d19c3d0a P 17062e38120042aaaca05e153b2c6ccf [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:02.491420 14548 raft_consensus.cc:3037] T acd35bd40253406fa1baa3f0d19c3d0a P e625ab1b03d14e2aae194052d2c67bc6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:02.492162 14417 raft_consensus.cc:2454] T acd35bd40253406fa1baa3f0d19c3d0a P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c87cbbf2d3ab4143815ef2c74ae760db in term 1.
I20240501 14:16:02.492162 14548 raft_consensus.cc:2454] T acd35bd40253406fa1baa3f0d19c3d0a P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c87cbbf2d3ab4143815ef2c74ae760db in term 1.
I20240501 14:16:02.492405 14222 leader_election.cc:302] T acd35bd40253406fa1baa3f0d19c3d0a P c87cbbf2d3ab4143815ef2c74ae760db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c87cbbf2d3ab4143815ef2c74ae760db, e625ab1b03d14e2aae194052d2c67bc6; no voters:
I20240501 14:16:02.492585 14897 raft_consensus.cc:2781] T acd35bd40253406fa1baa3f0d19c3d0a P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:16:02.492666 14897 raft_consensus.cc:686] T acd35bd40253406fa1baa3f0d19c3d0a P c87cbbf2d3ab4143815ef2c74ae760db [term 1 LEADER]: Becoming Leader. State: Replica: c87cbbf2d3ab4143815ef2c74ae760db, State: Running, Role: LEADER
I20240501 14:16:02.492805 14897 consensus_queue.cc:237] T acd35bd40253406fa1baa3f0d19c3d0a P c87cbbf2d3ab4143815ef2c74ae760db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } }
I20240501 14:16:02.494169 14136 catalog_manager.cc:5484] T acd35bd40253406fa1baa3f0d19c3d0a P c87cbbf2d3ab4143815ef2c74ae760db reported cstate change: term changed from 0 to 1, leader changed from <none> to c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129). New cstate: current_term: 1 leader_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } health_report { overall_health: UNKNOWN } } }
W20240501 14:16:02.644186 15026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:16:02.644407 15026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:16:02.647137 15026 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240501 14:16:02.730993 14548 raft_consensus.cc:1261] T 00618ea639cf49d09cc3523b47cd0c07 P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Refusing update from remote peer 17062e38120042aaaca05e153b2c6ccf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:16:02.731539 14901 consensus_queue.cc:1035] T 00618ea639cf49d09cc3523b47cd0c07 P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:16:02.731866 14285 raft_consensus.cc:1261] T 00618ea639cf49d09cc3523b47cd0c07 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Refusing update from remote peer 17062e38120042aaaca05e153b2c6ccf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:16:02.732296 14901 consensus_queue.cc:1035] T 00618ea639cf49d09cc3523b47cd0c07 P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:16:02.739112 14286 raft_consensus.cc:1261] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Refusing update from remote peer e625ab1b03d14e2aae194052d2c67bc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:16:02.739506 14975 consensus_queue.cc:1035] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:16:02.740634 14417 raft_consensus.cc:1261] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Refusing update from remote peer e625ab1b03d14e2aae194052d2c67bc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:16:02.741032 14975 consensus_queue.cc:1035] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:16:02.744886 14418 raft_consensus.cc:1261] T acd35bd40253406fa1baa3f0d19c3d0a P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Refusing update from remote peer c87cbbf2d3ab4143815ef2c74ae760db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:16:02.747316 14897 consensus_queue.cc:1035] T acd35bd40253406fa1baa3f0d19c3d0a P c87cbbf2d3ab4143815ef2c74ae760db [LEADER]: Connected to new peer: Peer: permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:16:02.750996 14548 raft_consensus.cc:1261] T acd35bd40253406fa1baa3f0d19c3d0a P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Refusing update from remote peer c87cbbf2d3ab4143815ef2c74ae760db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:16:02.752501 14897 consensus_queue.cc:1035] T acd35bd40253406fa1baa3f0d19c3d0a P c87cbbf2d3ab4143815ef2c74ae760db [LEADER]: Connected to new peer: Peer: permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 122.86 ms
time per row: 0.06143 ms
I20240501 14:16:02.808887 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.133:36729
--local_ip_for_outbound_sockets=127.7.39.133
--tserver_master_addrs=127.7.39.190:36791
--webserver_port=45277
--webserver_interface=127.7.39.133
--builtin_ntp_servers=127.7.39.148:34331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 14:16:02.912509 15086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:16:02.912799 15086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:16:02.912880 15086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:16:02.915944 15086 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:16:02.916008 15086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:16:02.916100 15086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.133
I20240501 14:16:02.919518 15086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:34331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.133:36729
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.7.39.133
--webserver_port=45277
--tserver_master_addrs=127.7.39.190:36791
--never_fsync=true
--heap_profile_path=/tmp/kudu.15086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.133
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:16:02.920332 15086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:16:02.921454 15086 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:16:02.930207 15091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:16:02.930801 15092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:16:02.931366 15094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:16:02.932078 15086 server_base.cc:1034] running on GCE node
I20240501 14:16:02.932448 15086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:16:02.933001 15086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:16:02.934156 15086 hybrid_clock.cc:648] HybridClock initialized: now 1714572962934137 us; error 38 us; skew 500 ppm
I20240501 14:16:02.935899 15086 webserver.cc:460] Webserver started at http://127.7.39.133:45277/ using document root <none> and password file <none>
I20240501 14:16:02.936447 15086 fs_manager.cc:359] Metadata directory not provided
I20240501 14:16:02.936525 15086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:16:02.939998 15086 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:16:02.941802 15100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:16:02.942826 15086 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:16:02.942953 15086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "50161459714942cb9b4ae7743f76d07d"
format_stamp: "Formatted at 2024-05-01 14:15:57 on dist-test-slave-rnfw"
I20240501 14:16:02.943449 15086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:16:02.967588 15086 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:16:02.968252 15086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:16:02.968434 15086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:16:02.968976 15086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:16:02.970063 15086 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:16:02.970124 15086 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:16:02.970196 15086 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:16:02.970223 15086 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:16:02.978994 15086 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.133:36729
I20240501 14:16:02.979020 15213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.133:36729 every 8 connection(s)
I20240501 14:16:02.980024 15086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 14:16:02.984809 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 15086
I20240501 14:16:02.985248 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.132:38927
--local_ip_for_outbound_sockets=127.7.39.132
--tserver_master_addrs=127.7.39.190:36791
--webserver_port=39171
--webserver_interface=127.7.39.132
--builtin_ntp_servers=127.7.39.148:34331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:16:02.988946 15214 heartbeater.cc:347] Connected to a master server at 127.7.39.190:36791
I20240501 14:16:02.989166 15214 heartbeater.cc:464] Registering TS with master...
I20240501 14:16:02.989660 15214 heartbeater.cc:511] Master 127.7.39.190:36791 requested a full tablet report, sending...
I20240501 14:16:02.990841 14141 ts_manager.cc:194] Re-registered known tserver with Master: 50161459714942cb9b4ae7743f76d07d (127.7.39.133:36729)
I20240501 14:16:02.991458 14141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.133:54849
W20240501 14:16:03.090920 15217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:16:03.091157 15217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:16:03.091215 15217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:16:03.094132 15217 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:16:03.094194 15217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:16:03.094281 15217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.132
I20240501 14:16:03.097903 15217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:34331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.132:38927
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.7.39.132
--webserver_port=39171
--tserver_master_addrs=127.7.39.190:36791
--never_fsync=true
--heap_profile_path=/tmp/kudu.15217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.132
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:16:03.098744 15217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:16:03.099853 15217 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:16:03.108350 15223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:16:03.108824 15225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:16:03.109548 15222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:16:03.110235 15217 server_base.cc:1034] running on GCE node
I20240501 14:16:03.110630 15217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:16:03.111277 15217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:16:03.112439 15217 hybrid_clock.cc:648] HybridClock initialized: now 1714572963112430 us; error 29 us; skew 500 ppm
I20240501 14:16:03.114480 15217 webserver.cc:460] Webserver started at http://127.7.39.132:39171/ using document root <none> and password file <none>
I20240501 14:16:03.115022 15217 fs_manager.cc:359] Metadata directory not provided
I20240501 14:16:03.115149 15217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:16:03.118465 15217 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:16:03.120406 15231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:16:03.121426 15217 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 14:16:03.121533 15217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "e3770e9581534f77a7506253b8d201b0"
format_stamp: "Formatted at 2024-05-01 14:15:57 on dist-test-slave-rnfw"
I20240501 14:16:03.121919 15217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:16:03.136286 15217 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:16:03.136773 15217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:16:03.136926 15217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:16:03.137404 15217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:16:03.138221 15217 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:16:03.138268 15217 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:16:03.138311 15217 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:16:03.138343 15217 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:16:03.147442 15217 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.132:38927
I20240501 14:16:03.147490 15344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.132:38927 every 8 connection(s)
I20240501 14:16:03.148432 15217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 14:16:03.151845 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 15217
I20240501 14:16:03.157430 15345 heartbeater.cc:347] Connected to a master server at 127.7.39.190:36791
I20240501 14:16:03.157621 15345 heartbeater.cc:464] Registering TS with master...
I20240501 14:16:03.158083 15345 heartbeater.cc:511] Master 127.7.39.190:36791 requested a full tablet report, sending...
I20240501 14:16:03.159113 14141 ts_manager.cc:194] Re-registered known tserver with Master: e3770e9581534f77a7506253b8d201b0 (127.7.39.132:38927)
I20240501 14:16:03.159668 14141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.132:53687
I20240501 14:16:03.341384 14397 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:16:03.342670 15148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:16:03.344087 14264 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:16:03.344949 14528 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:16:03.352999 15279 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:16:03.436993 14397 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:16:03.438516 15148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:16:03.438550 14264 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:16:03.438899 15279 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:16:03.447471 14528 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:16:03.543486 15148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:16:03.544142 14397 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:16:03.546012 14264 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:16:03.551815 14528 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:16:03.560074 15279 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:16:03.635929 14264 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:16:03.638314 15148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:16:03.638334 14397 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:16:03.639026 14528 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:16:03.639597 15279 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:16:03.681070 14548 consensus_queue.cc:237] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: true } }
I20240501 14:16:03.682619 14286 raft_consensus.cc:1261] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Refusing update from remote peer e625ab1b03d14e2aae194052d2c67bc6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:16:03.682965 14418 raft_consensus.cc:1261] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Refusing update from remote peer e625ab1b03d14e2aae194052d2c67bc6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:16:03.683012 14885 consensus_queue.cc:1035] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:16:03.683405 14885 consensus_queue.cc:1035] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:16:03.684845 15493 raft_consensus.cc:2932] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 50161459714942cb9b4ae7743f76d07d (127.7.39.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: true } } }
I20240501 14:16:03.685220 14286 raft_consensus.cc:2932] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 50161459714942cb9b4ae7743f76d07d (127.7.39.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: true } } }
I20240501 14:16:03.687619 14418 raft_consensus.cc:2932] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 50161459714942cb9b4ae7743f76d07d (127.7.39.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: true } } }
I20240501 14:16:03.687924 14141 catalog_manager.cc:5484] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6 reported cstate change: config changed from index -1 to 4, NON_VOTER 50161459714942cb9b4ae7743f76d07d (127.7.39.133) added. New cstate: current_term: 1 leader_uuid: "e625ab1b03d14e2aae194052d2c67bc6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 14:16:03.688889 14485 consensus_peers.cc:485] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6 -> Peer 50161459714942cb9b4ae7743f76d07d (127.7.39.133:36729): Couldn't send request to peer 50161459714942cb9b4ae7743f76d07d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 47ef7ee4ea3b46cbbac2126ceaa5fa24. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:16:03.694931 14418 consensus_queue.cc:237] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: NON_VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: true } }
I20240501 14:16:03.696452 14286 raft_consensus.cc:1261] T 5f2b92ee1a1c46bcb36bc24563886d18 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Refusing update from remote peer 17062e38120042aaaca05e153b2c6ccf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:16:03.696648 14548 raft_consensus.cc:1261] T 5f2b92ee1a1c46bcb36bc24563886d18 P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Refusing update from remote peer 17062e38120042aaaca05e153b2c6ccf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:16:03.696933 15505 consensus_queue.cc:1035] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:16:03.698669 15505 consensus_queue.cc:1035] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:16:03.699687 15505 raft_consensus.cc:2932] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e3770e9581534f77a7506253b8d201b0 (127.7.39.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: NON_VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: true } } }
W20240501 14:16:03.701164 14351 consensus_peers.cc:485] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf -> Peer e3770e9581534f77a7506253b8d201b0 (127.7.39.132:38927): Couldn't send request to peer e3770e9581534f77a7506253b8d201b0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5f2b92ee1a1c46bcb36bc24563886d18. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:16:03.701252 14286 raft_consensus.cc:2932] T 5f2b92ee1a1c46bcb36bc24563886d18 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e3770e9581534f77a7506253b8d201b0 (127.7.39.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: NON_VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: true } } }
I20240501 14:16:03.701792 14548 raft_consensus.cc:2932] T 5f2b92ee1a1c46bcb36bc24563886d18 P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e3770e9581534f77a7506253b8d201b0 (127.7.39.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: NON_VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: true } } }
I20240501 14:16:03.703807 14136 catalog_manager.cc:5484] T 5f2b92ee1a1c46bcb36bc24563886d18 P c87cbbf2d3ab4143815ef2c74ae760db reported cstate change: config changed from index -1 to 4, NON_VOTER e3770e9581534f77a7506253b8d201b0 (127.7.39.132) added. New cstate: current_term: 1 leader_uuid: "17062e38120042aaaca05e153b2c6ccf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: NON_VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: true } } }
I20240501 14:16:03.708627 14286 consensus_queue.cc:237] T ca266243bcb84e8c9b65d1b096b5725d P c87cbbf2d3ab4143815ef2c74ae760db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: true } }
I20240501 14:16:03.709834 14418 raft_consensus.cc:1261] T ca266243bcb84e8c9b65d1b096b5725d P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Refusing update from remote peer c87cbbf2d3ab4143815ef2c74ae760db: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:16:03.709930 14548 raft_consensus.cc:1261] T ca266243bcb84e8c9b65d1b096b5725d P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Refusing update from remote peer c87cbbf2d3ab4143815ef2c74ae760db: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:16:03.710183 14897 consensus_queue.cc:1035] T ca266243bcb84e8c9b65d1b096b5725d P c87cbbf2d3ab4143815ef2c74ae760db [LEADER]: Connected to new peer: Peer: permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:16:03.710387 15516 consensus_queue.cc:1035] T ca266243bcb84e8c9b65d1b096b5725d P c87cbbf2d3ab4143815ef2c74ae760db [LEADER]: Connected to new peer: Peer: permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:16:03.712394 14897 raft_consensus.cc:2932] T ca266243bcb84e8c9b65d1b096b5725d P c87cbbf2d3ab4143815ef2c74ae760db [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 50161459714942cb9b4ae7743f76d07d (127.7.39.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: true } } }
I20240501 14:16:03.713181 14418 raft_consensus.cc:2932] T ca266243bcb84e8c9b65d1b096b5725d P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 50161459714942cb9b4ae7743f76d07d (127.7.39.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: true } } }
W20240501 14:16:03.713416 14221 consensus_peers.cc:485] T ca266243bcb84e8c9b65d1b096b5725d P c87cbbf2d3ab4143815ef2c74ae760db -> Peer 50161459714942cb9b4ae7743f76d07d (127.7.39.133:36729): Couldn't send request to peer 50161459714942cb9b4ae7743f76d07d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ca266243bcb84e8c9b65d1b096b5725d. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:16:03.714794 14141 catalog_manager.cc:5484] T ca266243bcb84e8c9b65d1b096b5725d P c87cbbf2d3ab4143815ef2c74ae760db reported cstate change: config changed from index -1 to 4, NON_VOTER 50161459714942cb9b4ae7743f76d07d (127.7.39.133) added. New cstate: current_term: 1 leader_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:16:03.714951 14548 raft_consensus.cc:2932] T ca266243bcb84e8c9b65d1b096b5725d P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 50161459714942cb9b4ae7743f76d07d (127.7.39.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: true } } }
I20240501 14:16:03.721025 14548 consensus_queue.cc:237] T 8ebd7203c0914f6c8e26ab4d20074765 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: NON_VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: true } }
I20240501 14:16:03.722193 14418 raft_consensus.cc:1261] T 8ebd7203c0914f6c8e26ab4d20074765 P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Refusing update from remote peer e625ab1b03d14e2aae194052d2c67bc6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:16:03.722528 14885 consensus_queue.cc:1035] T 8ebd7203c0914f6c8e26ab4d20074765 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:16:03.722592 14286 raft_consensus.cc:1261] T 8ebd7203c0914f6c8e26ab4d20074765 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Refusing update from remote peer e625ab1b03d14e2aae194052d2c67bc6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:16:03.722967 14885 consensus_queue.cc:1035] T 8ebd7203c0914f6c8e26ab4d20074765 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:16:03.724579 15492 raft_consensus.cc:2932] T 8ebd7203c0914f6c8e26ab4d20074765 P e625ab1b03d14e2aae194052d2c67bc6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e3770e9581534f77a7506253b8d201b0 (127.7.39.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: NON_VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: true } } }
I20240501 14:16:03.724975 14418 raft_consensus.cc:2932] T 8ebd7203c0914f6c8e26ab4d20074765 P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e3770e9581534f77a7506253b8d201b0 (127.7.39.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: NON_VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: true } } }
W20240501 14:16:03.726039 14483 consensus_peers.cc:485] T 8ebd7203c0914f6c8e26ab4d20074765 P e625ab1b03d14e2aae194052d2c67bc6 -> Peer e3770e9581534f77a7506253b8d201b0 (127.7.39.132:38927): Couldn't send request to peer e3770e9581534f77a7506253b8d201b0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ebd7203c0914f6c8e26ab4d20074765. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:16:03.726442 14286 raft_consensus.cc:2932] T 8ebd7203c0914f6c8e26ab4d20074765 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e3770e9581534f77a7506253b8d201b0 (127.7.39.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: NON_VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: true } } }
I20240501 14:16:03.726923 14136 catalog_manager.cc:5484] T 8ebd7203c0914f6c8e26ab4d20074765 P e625ab1b03d14e2aae194052d2c67bc6 reported cstate change: config changed from index -1 to 4, NON_VOTER e3770e9581534f77a7506253b8d201b0 (127.7.39.132) added. New cstate: current_term: 1 leader_uuid: "e625ab1b03d14e2aae194052d2c67bc6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: NON_VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:16:03.739719 14418 consensus_queue.cc:237] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: true } }
I20240501 14:16:03.741482 14286 raft_consensus.cc:1261] T 40732c47173c429aa69c6a41475849de P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Refusing update from remote peer 17062e38120042aaaca05e153b2c6ccf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:16:03.741521 14548 raft_consensus.cc:1261] T 40732c47173c429aa69c6a41475849de P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Refusing update from remote peer 17062e38120042aaaca05e153b2c6ccf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:16:03.742036 15505 consensus_queue.cc:1035] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:16:03.742376 15505 consensus_queue.cc:1035] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:16:03.745258 14353 consensus_peers.cc:485] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf -> Peer 50161459714942cb9b4ae7743f76d07d (127.7.39.133:36729): Couldn't send request to peer 50161459714942cb9b4ae7743f76d07d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 40732c47173c429aa69c6a41475849de. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:16:03.745507 15073 raft_consensus.cc:2932] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 50161459714942cb9b4ae7743f76d07d (127.7.39.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: true } } }
I20240501 14:16:03.745945 14286 raft_consensus.cc:2932] T 40732c47173c429aa69c6a41475849de P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 50161459714942cb9b4ae7743f76d07d (127.7.39.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: true } } }
I20240501 14:16:03.747938 14141 catalog_manager.cc:5484] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf reported cstate change: config changed from index -1 to 4, NON_VOTER 50161459714942cb9b4ae7743f76d07d (127.7.39.133) added. New cstate: current_term: 1 leader_uuid: "17062e38120042aaaca05e153b2c6ccf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:16:03.755163 14286 consensus_queue.cc:237] T acd35bd40253406fa1baa3f0d19c3d0a P c87cbbf2d3ab4143815ef2c74ae760db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: true } }
I20240501 14:16:03.756284 14549 raft_consensus.cc:1261] T acd35bd40253406fa1baa3f0d19c3d0a P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Refusing update from remote peer c87cbbf2d3ab4143815ef2c74ae760db: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:16:03.756557 14418 raft_consensus.cc:1261] T acd35bd40253406fa1baa3f0d19c3d0a P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Refusing update from remote peer c87cbbf2d3ab4143815ef2c74ae760db: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:16:03.756681 15517 consensus_queue.cc:1035] T acd35bd40253406fa1baa3f0d19c3d0a P c87cbbf2d3ab4143815ef2c74ae760db [LEADER]: Connected to new peer: Peer: permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:16:03.756822 14221 consensus_peers.cc:485] T acd35bd40253406fa1baa3f0d19c3d0a P c87cbbf2d3ab4143815ef2c74ae760db -> Peer 50161459714942cb9b4ae7743f76d07d (127.7.39.133:36729): Couldn't send request to peer 50161459714942cb9b4ae7743f76d07d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: acd35bd40253406fa1baa3f0d19c3d0a. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:16:03.756904 15516 consensus_queue.cc:1035] T acd35bd40253406fa1baa3f0d19c3d0a P c87cbbf2d3ab4143815ef2c74ae760db [LEADER]: Connected to new peer: Peer: permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:16:03.758211 15521 raft_consensus.cc:2932] T acd35bd40253406fa1baa3f0d19c3d0a P c87cbbf2d3ab4143815ef2c74ae760db [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 50161459714942cb9b4ae7743f76d07d (127.7.39.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: true } } }
I20240501 14:16:03.758765 14548 raft_consensus.cc:2932] T acd35bd40253406fa1baa3f0d19c3d0a P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 50161459714942cb9b4ae7743f76d07d (127.7.39.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: true } } }
I20240501 14:16:03.760284 14418 raft_consensus.cc:2932] T acd35bd40253406fa1baa3f0d19c3d0a P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 50161459714942cb9b4ae7743f76d07d (127.7.39.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: true } } }
I20240501 14:16:03.760968 14136 catalog_manager.cc:5484] T acd35bd40253406fa1baa3f0d19c3d0a P e625ab1b03d14e2aae194052d2c67bc6 reported cstate change: config changed from index -1 to 4, NON_VOTER 50161459714942cb9b4ae7743f76d07d (127.7.39.133) added. New cstate: current_term: 1 leader_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: true } } }
I20240501 14:16:03.761379 14548 raft_consensus.cc:2932] T 40732c47173c429aa69c6a41475849de P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 50161459714942cb9b4ae7743f76d07d (127.7.39.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: true } } }
I20240501 14:16:03.769513 14418 consensus_queue.cc:237] T 00618ea639cf49d09cc3523b47cd0c07 P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: NON_VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: true } }
I20240501 14:16:03.770558 14548 raft_consensus.cc:1261] T 00618ea639cf49d09cc3523b47cd0c07 P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Refusing update from remote peer 17062e38120042aaaca05e153b2c6ccf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:16:03.770745 14351 consensus_peers.cc:485] T 00618ea639cf49d09cc3523b47cd0c07 P 17062e38120042aaaca05e153b2c6ccf -> Peer e3770e9581534f77a7506253b8d201b0 (127.7.39.132:38927): Couldn't send request to peer e3770e9581534f77a7506253b8d201b0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 00618ea639cf49d09cc3523b47cd0c07. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:16:03.770910 15073 consensus_queue.cc:1035] T 00618ea639cf49d09cc3523b47cd0c07 P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:16:03.771090 14286 raft_consensus.cc:1261] T 00618ea639cf49d09cc3523b47cd0c07 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Refusing update from remote peer 17062e38120042aaaca05e153b2c6ccf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:16:03.771442 15073 consensus_queue.cc:1035] T 00618ea639cf49d09cc3523b47cd0c07 P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:16:03.772406 15506 raft_consensus.cc:2932] T 00618ea639cf49d09cc3523b47cd0c07 P 17062e38120042aaaca05e153b2c6ccf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e3770e9581534f77a7506253b8d201b0 (127.7.39.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: NON_VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: true } } }
I20240501 14:16:03.772903 14548 raft_consensus.cc:2932] T 00618ea639cf49d09cc3523b47cd0c07 P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e3770e9581534f77a7506253b8d201b0 (127.7.39.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: NON_VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: true } } }
I20240501 14:16:03.773317 14286 raft_consensus.cc:2932] T 00618ea639cf49d09cc3523b47cd0c07 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e3770e9581534f77a7506253b8d201b0 (127.7.39.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: NON_VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: true } } }
I20240501 14:16:03.775389 14136 catalog_manager.cc:5484] T 00618ea639cf49d09cc3523b47cd0c07 P e625ab1b03d14e2aae194052d2c67bc6 reported cstate change: config changed from index -1 to 4, NON_VOTER e3770e9581534f77a7506253b8d201b0 (127.7.39.132) added. New cstate: current_term: 1 leader_uuid: "17062e38120042aaaca05e153b2c6ccf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: NON_VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: true } } }
I20240501 14:16:03.993355 15214 heartbeater.cc:503] Master 127.7.39.190:36791 was elected leader, sending a full tablet report...
I20240501 14:16:04.148578 15618 ts_tablet_manager.cc:902] T 00618ea639cf49d09cc3523b47cd0c07 P e3770e9581534f77a7506253b8d201b0: Initiating tablet copy from peer 17062e38120042aaaca05e153b2c6ccf (127.7.39.130:43603)
I20240501 14:16:04.149299 15618 tablet_copy_client.cc:308] T 00618ea639cf49d09cc3523b47cd0c07 P e3770e9581534f77a7506253b8d201b0: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.130:43603
I20240501 14:16:04.150815 15624 ts_tablet_manager.cc:902] T ca266243bcb84e8c9b65d1b096b5725d P 50161459714942cb9b4ae7743f76d07d: Initiating tablet copy from peer c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129:42369)
I20240501 14:16:04.151618 15624 tablet_copy_client.cc:308] T ca266243bcb84e8c9b65d1b096b5725d P 50161459714942cb9b4ae7743f76d07d: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.129:42369
I20240501 14:16:04.154119 14438 tablet_copy_service.cc:140] P 17062e38120042aaaca05e153b2c6ccf: Received BeginTabletCopySession request for tablet 00618ea639cf49d09cc3523b47cd0c07 from peer e3770e9581534f77a7506253b8d201b0 ({username='slave'} at 127.7.39.132:48275)
I20240501 14:16:04.154273 14438 tablet_copy_service.cc:161] P 17062e38120042aaaca05e153b2c6ccf: Beginning new tablet copy session on tablet 00618ea639cf49d09cc3523b47cd0c07 from peer e3770e9581534f77a7506253b8d201b0 at {username='slave'} at 127.7.39.132:48275: session id = e3770e9581534f77a7506253b8d201b0-00618ea639cf49d09cc3523b47cd0c07
I20240501 14:16:04.155941 14438 tablet_copy_source_session.cc:205] T 00618ea639cf49d09cc3523b47cd0c07 P 17062e38120042aaaca05e153b2c6ccf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:16:04.155993 14306 tablet_copy_service.cc:140] P c87cbbf2d3ab4143815ef2c74ae760db: Received BeginTabletCopySession request for tablet ca266243bcb84e8c9b65d1b096b5725d from peer 50161459714942cb9b4ae7743f76d07d ({username='slave'} at 127.7.39.133:55727)
I20240501 14:16:04.156134 14306 tablet_copy_service.cc:161] P c87cbbf2d3ab4143815ef2c74ae760db: Beginning new tablet copy session on tablet ca266243bcb84e8c9b65d1b096b5725d from peer 50161459714942cb9b4ae7743f76d07d at {username='slave'} at 127.7.39.133:55727: session id = 50161459714942cb9b4ae7743f76d07d-ca266243bcb84e8c9b65d1b096b5725d
I20240501 14:16:04.157341 14306 tablet_copy_source_session.cc:205] T ca266243bcb84e8c9b65d1b096b5725d P c87cbbf2d3ab4143815ef2c74ae760db: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:16:04.159431 15618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00618ea639cf49d09cc3523b47cd0c07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:04.160200 15624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca266243bcb84e8c9b65d1b096b5725d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:04.161370 15345 heartbeater.cc:503] Master 127.7.39.190:36791 was elected leader, sending a full tablet report...
I20240501 14:16:04.165951 15618 tablet_copy_client.cc:783] T 00618ea639cf49d09cc3523b47cd0c07 P e3770e9581534f77a7506253b8d201b0: tablet copy: Starting download of 0 data blocks...
I20240501 14:16:04.166147 15618 tablet_copy_client.cc:647] T 00618ea639cf49d09cc3523b47cd0c07 P e3770e9581534f77a7506253b8d201b0: tablet copy: Starting download of 1 WAL segments...
I20240501 14:16:04.166188 15624 tablet_copy_client.cc:783] T ca266243bcb84e8c9b65d1b096b5725d P 50161459714942cb9b4ae7743f76d07d: tablet copy: Starting download of 0 data blocks...
I20240501 14:16:04.166385 15624 tablet_copy_client.cc:647] T ca266243bcb84e8c9b65d1b096b5725d P 50161459714942cb9b4ae7743f76d07d: tablet copy: Starting download of 1 WAL segments...
I20240501 14:16:04.167795 15618 tablet_copy_client.cc:523] T 00618ea639cf49d09cc3523b47cd0c07 P e3770e9581534f77a7506253b8d201b0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:16:04.167912 15624 tablet_copy_client.cc:523] T ca266243bcb84e8c9b65d1b096b5725d P 50161459714942cb9b4ae7743f76d07d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:16:04.169196 15624 tablet_bootstrap.cc:492] T ca266243bcb84e8c9b65d1b096b5725d P 50161459714942cb9b4ae7743f76d07d: Bootstrap starting.
I20240501 14:16:04.169195 15618 tablet_bootstrap.cc:492] T 00618ea639cf49d09cc3523b47cd0c07 P e3770e9581534f77a7506253b8d201b0: Bootstrap starting.
I20240501 14:16:04.174342 15624 log.cc:824] T ca266243bcb84e8c9b65d1b096b5725d P 50161459714942cb9b4ae7743f76d07d: Log is configured to *not* fsync() on all Append() calls
I20240501 14:16:04.174811 15618 log.cc:824] T 00618ea639cf49d09cc3523b47cd0c07 P e3770e9581534f77a7506253b8d201b0: Log is configured to *not* fsync() on all Append() calls
I20240501 14:16:04.194049 15624 tablet_bootstrap.cc:492] T ca266243bcb84e8c9b65d1b096b5725d P 50161459714942cb9b4ae7743f76d07d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:16:04.194386 15624 tablet_bootstrap.cc:492] T ca266243bcb84e8c9b65d1b096b5725d P 50161459714942cb9b4ae7743f76d07d: Bootstrap complete.
I20240501 14:16:04.194669 15624 ts_tablet_manager.cc:1367] T ca266243bcb84e8c9b65d1b096b5725d P 50161459714942cb9b4ae7743f76d07d: Time spent bootstrapping tablet: real 0.026s user 0.019s sys 0.004s
I20240501 14:16:04.196208 15618 tablet_bootstrap.cc:492] T 00618ea639cf49d09cc3523b47cd0c07 P e3770e9581534f77a7506253b8d201b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:16:04.196523 15618 tablet_bootstrap.cc:492] T 00618ea639cf49d09cc3523b47cd0c07 P e3770e9581534f77a7506253b8d201b0: Bootstrap complete.
I20240501 14:16:04.196604 15624 raft_consensus.cc:348] T ca266243bcb84e8c9b65d1b096b5725d P 50161459714942cb9b4ae7743f76d07d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: true } }
I20240501 14:16:04.196787 15618 ts_tablet_manager.cc:1367] T 00618ea639cf49d09cc3523b47cd0c07 P e3770e9581534f77a7506253b8d201b0: Time spent bootstrapping tablet: real 0.028s user 0.019s sys 0.008s
I20240501 14:16:04.196820 15624 raft_consensus.cc:729] T ca266243bcb84e8c9b65d1b096b5725d P 50161459714942cb9b4ae7743f76d07d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 50161459714942cb9b4ae7743f76d07d, State: Initialized, Role: LEARNER
I20240501 14:16:04.197118 15624 consensus_queue.cc:260] T ca266243bcb84e8c9b65d1b096b5725d P 50161459714942cb9b4ae7743f76d07d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: true } }
I20240501 14:16:04.197897 15624 ts_tablet_manager.cc:1398] T ca266243bcb84e8c9b65d1b096b5725d P 50161459714942cb9b4ae7743f76d07d: Time spent starting tablet: real 0.003s user 0.006s sys 0.000s
I20240501 14:16:04.198654 14306 tablet_copy_service.cc:342] P c87cbbf2d3ab4143815ef2c74ae760db: Request end of tablet copy session 50161459714942cb9b4ae7743f76d07d-ca266243bcb84e8c9b65d1b096b5725d received from {username='slave'} at 127.7.39.133:55727
I20240501 14:16:04.198767 14306 tablet_copy_service.cc:433] P c87cbbf2d3ab4143815ef2c74ae760db: ending tablet copy session 50161459714942cb9b4ae7743f76d07d-ca266243bcb84e8c9b65d1b096b5725d on tablet ca266243bcb84e8c9b65d1b096b5725d with peer 50161459714942cb9b4ae7743f76d07d
I20240501 14:16:04.198933 15618 raft_consensus.cc:348] T 00618ea639cf49d09cc3523b47cd0c07 P e3770e9581534f77a7506253b8d201b0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: NON_VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: true } }
I20240501 14:16:04.199189 15618 raft_consensus.cc:729] T 00618ea639cf49d09cc3523b47cd0c07 P e3770e9581534f77a7506253b8d201b0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e3770e9581534f77a7506253b8d201b0, State: Initialized, Role: LEARNER
I20240501 14:16:04.199535 15618 consensus_queue.cc:260] T 00618ea639cf49d09cc3523b47cd0c07 P e3770e9581534f77a7506253b8d201b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: NON_VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: true } }
I20240501 14:16:04.200419 15618 ts_tablet_manager.cc:1398] T 00618ea639cf49d09cc3523b47cd0c07 P e3770e9581534f77a7506253b8d201b0: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240501 14:16:04.201277 14438 tablet_copy_service.cc:342] P 17062e38120042aaaca05e153b2c6ccf: Request end of tablet copy session e3770e9581534f77a7506253b8d201b0-00618ea639cf49d09cc3523b47cd0c07 received from {username='slave'} at 127.7.39.132:48275
I20240501 14:16:04.201362 14438 tablet_copy_service.cc:433] P 17062e38120042aaaca05e153b2c6ccf: ending tablet copy session e3770e9581534f77a7506253b8d201b0-00618ea639cf49d09cc3523b47cd0c07 on tablet 00618ea639cf49d09cc3523b47cd0c07 with peer e3770e9581534f77a7506253b8d201b0
I20240501 14:16:04.215813 15624 ts_tablet_manager.cc:902] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 50161459714942cb9b4ae7743f76d07d: Initiating tablet copy from peer e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131:33199)
I20240501 14:16:04.216282 15624 tablet_copy_client.cc:308] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 50161459714942cb9b4ae7743f76d07d: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.131:33199
I20240501 14:16:04.219450 14570 tablet_copy_service.cc:140] P e625ab1b03d14e2aae194052d2c67bc6: Received BeginTabletCopySession request for tablet 47ef7ee4ea3b46cbbac2126ceaa5fa24 from peer 50161459714942cb9b4ae7743f76d07d ({username='slave'} at 127.7.39.133:58011)
I20240501 14:16:04.219610 14570 tablet_copy_service.cc:161] P e625ab1b03d14e2aae194052d2c67bc6: Beginning new tablet copy session on tablet 47ef7ee4ea3b46cbbac2126ceaa5fa24 from peer 50161459714942cb9b4ae7743f76d07d at {username='slave'} at 127.7.39.133:58011: session id = 50161459714942cb9b4ae7743f76d07d-47ef7ee4ea3b46cbbac2126ceaa5fa24
I20240501 14:16:04.219996 15618 ts_tablet_manager.cc:902] T 5f2b92ee1a1c46bcb36bc24563886d18 P e3770e9581534f77a7506253b8d201b0: Initiating tablet copy from peer 17062e38120042aaaca05e153b2c6ccf (127.7.39.130:43603)
I20240501 14:16:04.220453 15618 tablet_copy_client.cc:308] T 5f2b92ee1a1c46bcb36bc24563886d18 P e3770e9581534f77a7506253b8d201b0: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.130:43603
I20240501 14:16:04.220916 14438 tablet_copy_service.cc:140] P 17062e38120042aaaca05e153b2c6ccf: Received BeginTabletCopySession request for tablet 5f2b92ee1a1c46bcb36bc24563886d18 from peer e3770e9581534f77a7506253b8d201b0 ({username='slave'} at 127.7.39.132:48275)
I20240501 14:16:04.221050 14438 tablet_copy_service.cc:161] P 17062e38120042aaaca05e153b2c6ccf: Beginning new tablet copy session on tablet 5f2b92ee1a1c46bcb36bc24563886d18 from peer e3770e9581534f77a7506253b8d201b0 at {username='slave'} at 127.7.39.132:48275: session id = e3770e9581534f77a7506253b8d201b0-5f2b92ee1a1c46bcb36bc24563886d18
I20240501 14:16:04.221068 14570 tablet_copy_source_session.cc:205] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:16:04.221936 14438 tablet_copy_source_session.cc:205] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:16:04.222249 15624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47ef7ee4ea3b46cbbac2126ceaa5fa24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:04.222695 15618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f2b92ee1a1c46bcb36bc24563886d18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:04.224364 15624 tablet_copy_client.cc:783] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 50161459714942cb9b4ae7743f76d07d: tablet copy: Starting download of 0 data blocks...
I20240501 14:16:04.224501 15618 tablet_copy_client.cc:783] T 5f2b92ee1a1c46bcb36bc24563886d18 P e3770e9581534f77a7506253b8d201b0: tablet copy: Starting download of 0 data blocks...
I20240501 14:16:04.224514 15624 tablet_copy_client.cc:647] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 50161459714942cb9b4ae7743f76d07d: tablet copy: Starting download of 1 WAL segments...
I20240501 14:16:04.224654 15618 tablet_copy_client.cc:647] T 5f2b92ee1a1c46bcb36bc24563886d18 P e3770e9581534f77a7506253b8d201b0: tablet copy: Starting download of 1 WAL segments...
I20240501 14:16:04.225783 15618 tablet_copy_client.cc:523] T 5f2b92ee1a1c46bcb36bc24563886d18 P e3770e9581534f77a7506253b8d201b0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:16:04.226050 15624 tablet_copy_client.cc:523] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 50161459714942cb9b4ae7743f76d07d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:16:04.226924 15642 ts_tablet_manager.cc:902] T 8ebd7203c0914f6c8e26ab4d20074765 P e3770e9581534f77a7506253b8d201b0: Initiating tablet copy from peer e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131:33199)
I20240501 14:16:04.227203 15624 tablet_bootstrap.cc:492] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 50161459714942cb9b4ae7743f76d07d: Bootstrap starting.
I20240501 14:16:04.227250 15618 tablet_bootstrap.cc:492] T 5f2b92ee1a1c46bcb36bc24563886d18 P e3770e9581534f77a7506253b8d201b0: Bootstrap starting.
I20240501 14:16:04.227635 15642 tablet_copy_client.cc:308] T 8ebd7203c0914f6c8e26ab4d20074765 P e3770e9581534f77a7506253b8d201b0: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.131:33199
I20240501 14:16:04.232204 14570 tablet_copy_service.cc:140] P e625ab1b03d14e2aae194052d2c67bc6: Received BeginTabletCopySession request for tablet 8ebd7203c0914f6c8e26ab4d20074765 from peer e3770e9581534f77a7506253b8d201b0 ({username='slave'} at 127.7.39.132:42685)
I20240501 14:16:04.232308 14570 tablet_copy_service.cc:161] P e625ab1b03d14e2aae194052d2c67bc6: Beginning new tablet copy session on tablet 8ebd7203c0914f6c8e26ab4d20074765 from peer e3770e9581534f77a7506253b8d201b0 at {username='slave'} at 127.7.39.132:42685: session id = e3770e9581534f77a7506253b8d201b0-8ebd7203c0914f6c8e26ab4d20074765
I20240501 14:16:04.233198 14570 tablet_copy_source_session.cc:205] T 8ebd7203c0914f6c8e26ab4d20074765 P e625ab1b03d14e2aae194052d2c67bc6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:16:04.234030 15642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ebd7203c0914f6c8e26ab4d20074765. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:04.236001 15642 tablet_copy_client.cc:783] T 8ebd7203c0914f6c8e26ab4d20074765 P e3770e9581534f77a7506253b8d201b0: tablet copy: Starting download of 0 data blocks...
I20240501 14:16:04.236196 15642 tablet_copy_client.cc:647] T 8ebd7203c0914f6c8e26ab4d20074765 P e3770e9581534f77a7506253b8d201b0: tablet copy: Starting download of 1 WAL segments...
I20240501 14:16:04.237381 15642 tablet_copy_client.cc:523] T 8ebd7203c0914f6c8e26ab4d20074765 P e3770e9581534f77a7506253b8d201b0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:16:04.238538 15642 tablet_bootstrap.cc:492] T 8ebd7203c0914f6c8e26ab4d20074765 P e3770e9581534f77a7506253b8d201b0: Bootstrap starting.
I20240501 14:16:04.245795 15624 tablet_bootstrap.cc:492] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 50161459714942cb9b4ae7743f76d07d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:16:04.246071 15624 tablet_bootstrap.cc:492] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 50161459714942cb9b4ae7743f76d07d: Bootstrap complete.
I20240501 14:16:04.246261 15624 ts_tablet_manager.cc:1367] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 50161459714942cb9b4ae7743f76d07d: Time spent bootstrapping tablet: real 0.019s user 0.017s sys 0.000s
I20240501 14:16:04.247821 15624 raft_consensus.cc:348] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 50161459714942cb9b4ae7743f76d07d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: true } }
I20240501 14:16:04.247952 15624 raft_consensus.cc:729] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 50161459714942cb9b4ae7743f76d07d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 50161459714942cb9b4ae7743f76d07d, State: Initialized, Role: LEARNER
I20240501 14:16:04.248050 15624 consensus_queue.cc:260] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 50161459714942cb9b4ae7743f76d07d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: true } }
I20240501 14:16:04.248807 15624 ts_tablet_manager.cc:1398] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 50161459714942cb9b4ae7743f76d07d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:16:04.249336 14570 tablet_copy_service.cc:342] P e625ab1b03d14e2aae194052d2c67bc6: Request end of tablet copy session 50161459714942cb9b4ae7743f76d07d-47ef7ee4ea3b46cbbac2126ceaa5fa24 received from {username='slave'} at 127.7.39.133:58011
I20240501 14:16:04.249429 14570 tablet_copy_service.cc:433] P e625ab1b03d14e2aae194052d2c67bc6: ending tablet copy session 50161459714942cb9b4ae7743f76d07d-47ef7ee4ea3b46cbbac2126ceaa5fa24 on tablet 47ef7ee4ea3b46cbbac2126ceaa5fa24 with peer 50161459714942cb9b4ae7743f76d07d
I20240501 14:16:04.253355 15618 tablet_bootstrap.cc:492] T 5f2b92ee1a1c46bcb36bc24563886d18 P e3770e9581534f77a7506253b8d201b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:16:04.253584 15618 tablet_bootstrap.cc:492] T 5f2b92ee1a1c46bcb36bc24563886d18 P e3770e9581534f77a7506253b8d201b0: Bootstrap complete.
I20240501 14:16:04.253741 15618 ts_tablet_manager.cc:1367] T 5f2b92ee1a1c46bcb36bc24563886d18 P e3770e9581534f77a7506253b8d201b0: Time spent bootstrapping tablet: real 0.027s user 0.023s sys 0.000s
I20240501 14:16:04.254251 15618 raft_consensus.cc:348] T 5f2b92ee1a1c46bcb36bc24563886d18 P e3770e9581534f77a7506253b8d201b0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: NON_VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: true } }
I20240501 14:16:04.254348 15618 raft_consensus.cc:729] T 5f2b92ee1a1c46bcb36bc24563886d18 P e3770e9581534f77a7506253b8d201b0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e3770e9581534f77a7506253b8d201b0, State: Initialized, Role: LEARNER
I20240501 14:16:04.254454 15618 consensus_queue.cc:260] T 5f2b92ee1a1c46bcb36bc24563886d18 P e3770e9581534f77a7506253b8d201b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: NON_VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: true } }
I20240501 14:16:04.254701 15618 ts_tablet_manager.cc:1398] T 5f2b92ee1a1c46bcb36bc24563886d18 P e3770e9581534f77a7506253b8d201b0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:16:04.255378 14438 tablet_copy_service.cc:342] P 17062e38120042aaaca05e153b2c6ccf: Request end of tablet copy session e3770e9581534f77a7506253b8d201b0-5f2b92ee1a1c46bcb36bc24563886d18 received from {username='slave'} at 127.7.39.132:48275
I20240501 14:16:04.255463 14438 tablet_copy_service.cc:433] P 17062e38120042aaaca05e153b2c6ccf: ending tablet copy session e3770e9581534f77a7506253b8d201b0-5f2b92ee1a1c46bcb36bc24563886d18 on tablet 5f2b92ee1a1c46bcb36bc24563886d18 with peer e3770e9581534f77a7506253b8d201b0
I20240501 14:16:04.264508 15642 tablet_bootstrap.cc:492] T 8ebd7203c0914f6c8e26ab4d20074765 P e3770e9581534f77a7506253b8d201b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:16:04.264815 15642 tablet_bootstrap.cc:492] T 8ebd7203c0914f6c8e26ab4d20074765 P e3770e9581534f77a7506253b8d201b0: Bootstrap complete.
I20240501 14:16:04.264997 15642 ts_tablet_manager.cc:1367] T 8ebd7203c0914f6c8e26ab4d20074765 P e3770e9581534f77a7506253b8d201b0: Time spent bootstrapping tablet: real 0.026s user 0.015s sys 0.008s
I20240501 14:16:04.265372 15642 raft_consensus.cc:348] T 8ebd7203c0914f6c8e26ab4d20074765 P e3770e9581534f77a7506253b8d201b0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: NON_VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: true } }
I20240501 14:16:04.265507 15642 raft_consensus.cc:729] T 8ebd7203c0914f6c8e26ab4d20074765 P e3770e9581534f77a7506253b8d201b0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e3770e9581534f77a7506253b8d201b0, State: Initialized, Role: LEARNER
I20240501 14:16:04.265630 15642 consensus_queue.cc:260] T 8ebd7203c0914f6c8e26ab4d20074765 P e3770e9581534f77a7506253b8d201b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: NON_VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: true } }
I20240501 14:16:04.266053 15642 ts_tablet_manager.cc:1398] T 8ebd7203c0914f6c8e26ab4d20074765 P e3770e9581534f77a7506253b8d201b0: Time spent starting tablet: real 0.001s user 0.002s sys 0.001s
I20240501 14:16:04.266572 14570 tablet_copy_service.cc:342] P e625ab1b03d14e2aae194052d2c67bc6: Request end of tablet copy session e3770e9581534f77a7506253b8d201b0-8ebd7203c0914f6c8e26ab4d20074765 received from {username='slave'} at 127.7.39.132:42685
I20240501 14:16:04.266675 14570 tablet_copy_service.cc:433] P e625ab1b03d14e2aae194052d2c67bc6: ending tablet copy session e3770e9581534f77a7506253b8d201b0-8ebd7203c0914f6c8e26ab4d20074765 on tablet 8ebd7203c0914f6c8e26ab4d20074765 with peer e3770e9581534f77a7506253b8d201b0
I20240501 14:16:04.271708 15624 ts_tablet_manager.cc:902] T 40732c47173c429aa69c6a41475849de P 50161459714942cb9b4ae7743f76d07d: Initiating tablet copy from peer 17062e38120042aaaca05e153b2c6ccf (127.7.39.130:43603)
I20240501 14:16:04.272171 15624 tablet_copy_client.cc:308] T 40732c47173c429aa69c6a41475849de P 50161459714942cb9b4ae7743f76d07d: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.130:43603
I20240501 14:16:04.275884 14438 tablet_copy_service.cc:140] P 17062e38120042aaaca05e153b2c6ccf: Received BeginTabletCopySession request for tablet 40732c47173c429aa69c6a41475849de from peer 50161459714942cb9b4ae7743f76d07d ({username='slave'} at 127.7.39.133:41087)
I20240501 14:16:04.275975 14438 tablet_copy_service.cc:161] P 17062e38120042aaaca05e153b2c6ccf: Beginning new tablet copy session on tablet 40732c47173c429aa69c6a41475849de from peer 50161459714942cb9b4ae7743f76d07d at {username='slave'} at 127.7.39.133:41087: session id = 50161459714942cb9b4ae7743f76d07d-40732c47173c429aa69c6a41475849de
I20240501 14:16:04.277016 14438 tablet_copy_source_session.cc:205] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:16:04.277810 15624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40732c47173c429aa69c6a41475849de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:04.280014 15624 tablet_copy_client.cc:783] T 40732c47173c429aa69c6a41475849de P 50161459714942cb9b4ae7743f76d07d: tablet copy: Starting download of 0 data blocks...
I20240501 14:16:04.280162 15624 tablet_copy_client.cc:647] T 40732c47173c429aa69c6a41475849de P 50161459714942cb9b4ae7743f76d07d: tablet copy: Starting download of 1 WAL segments...
I20240501 14:16:04.281224 15624 tablet_copy_client.cc:523] T 40732c47173c429aa69c6a41475849de P 50161459714942cb9b4ae7743f76d07d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:16:04.282389 15624 tablet_bootstrap.cc:492] T 40732c47173c429aa69c6a41475849de P 50161459714942cb9b4ae7743f76d07d: Bootstrap starting.
I20240501 14:16:04.290767 15685 ts_tablet_manager.cc:902] T acd35bd40253406fa1baa3f0d19c3d0a P 50161459714942cb9b4ae7743f76d07d: Initiating tablet copy from peer c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129:42369)
I20240501 14:16:04.291369 15685 tablet_copy_client.cc:308] T acd35bd40253406fa1baa3f0d19c3d0a P 50161459714942cb9b4ae7743f76d07d: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.129:42369
I20240501 14:16:04.291832 14306 tablet_copy_service.cc:140] P c87cbbf2d3ab4143815ef2c74ae760db: Received BeginTabletCopySession request for tablet acd35bd40253406fa1baa3f0d19c3d0a from peer 50161459714942cb9b4ae7743f76d07d ({username='slave'} at 127.7.39.133:55727)
I20240501 14:16:04.291946 14306 tablet_copy_service.cc:161] P c87cbbf2d3ab4143815ef2c74ae760db: Beginning new tablet copy session on tablet acd35bd40253406fa1baa3f0d19c3d0a from peer 50161459714942cb9b4ae7743f76d07d at {username='slave'} at 127.7.39.133:55727: session id = 50161459714942cb9b4ae7743f76d07d-acd35bd40253406fa1baa3f0d19c3d0a
I20240501 14:16:04.293030 14306 tablet_copy_source_session.cc:205] T acd35bd40253406fa1baa3f0d19c3d0a P c87cbbf2d3ab4143815ef2c74ae760db: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:16:04.294133 15685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acd35bd40253406fa1baa3f0d19c3d0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:04.296474 15685 tablet_copy_client.cc:783] T acd35bd40253406fa1baa3f0d19c3d0a P 50161459714942cb9b4ae7743f76d07d: tablet copy: Starting download of 0 data blocks...
I20240501 14:16:04.296653 15685 tablet_copy_client.cc:647] T acd35bd40253406fa1baa3f0d19c3d0a P 50161459714942cb9b4ae7743f76d07d: tablet copy: Starting download of 1 WAL segments...
I20240501 14:16:04.298388 15685 tablet_copy_client.cc:523] T acd35bd40253406fa1baa3f0d19c3d0a P 50161459714942cb9b4ae7743f76d07d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:16:04.299343 15685 tablet_bootstrap.cc:492] T acd35bd40253406fa1baa3f0d19c3d0a P 50161459714942cb9b4ae7743f76d07d: Bootstrap starting.
I20240501 14:16:04.303436 15624 tablet_bootstrap.cc:492] T 40732c47173c429aa69c6a41475849de P 50161459714942cb9b4ae7743f76d07d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:16:04.303676 15624 tablet_bootstrap.cc:492] T 40732c47173c429aa69c6a41475849de P 50161459714942cb9b4ae7743f76d07d: Bootstrap complete.
I20240501 14:16:04.303834 15624 ts_tablet_manager.cc:1367] T 40732c47173c429aa69c6a41475849de P 50161459714942cb9b4ae7743f76d07d: Time spent bootstrapping tablet: real 0.022s user 0.020s sys 0.000s
I20240501 14:16:04.304178 15624 raft_consensus.cc:348] T 40732c47173c429aa69c6a41475849de P 50161459714942cb9b4ae7743f76d07d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: true } }
I20240501 14:16:04.304322 15624 raft_consensus.cc:729] T 40732c47173c429aa69c6a41475849de P 50161459714942cb9b4ae7743f76d07d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 50161459714942cb9b4ae7743f76d07d, State: Initialized, Role: LEARNER
I20240501 14:16:04.304417 15624 consensus_queue.cc:260] T 40732c47173c429aa69c6a41475849de P 50161459714942cb9b4ae7743f76d07d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: true } }
I20240501 14:16:04.304756 15624 ts_tablet_manager.cc:1398] T 40732c47173c429aa69c6a41475849de P 50161459714942cb9b4ae7743f76d07d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:16:04.305377 14438 tablet_copy_service.cc:342] P 17062e38120042aaaca05e153b2c6ccf: Request end of tablet copy session 50161459714942cb9b4ae7743f76d07d-40732c47173c429aa69c6a41475849de received from {username='slave'} at 127.7.39.133:41087
I20240501 14:16:04.305442 14438 tablet_copy_service.cc:433] P 17062e38120042aaaca05e153b2c6ccf: ending tablet copy session 50161459714942cb9b4ae7743f76d07d-40732c47173c429aa69c6a41475849de on tablet 40732c47173c429aa69c6a41475849de with peer 50161459714942cb9b4ae7743f76d07d
I20240501 14:16:04.319897 15685 tablet_bootstrap.cc:492] T acd35bd40253406fa1baa3f0d19c3d0a P 50161459714942cb9b4ae7743f76d07d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:16:04.320124 15685 tablet_bootstrap.cc:492] T acd35bd40253406fa1baa3f0d19c3d0a P 50161459714942cb9b4ae7743f76d07d: Bootstrap complete.
I20240501 14:16:04.320261 15685 ts_tablet_manager.cc:1367] T acd35bd40253406fa1baa3f0d19c3d0a P 50161459714942cb9b4ae7743f76d07d: Time spent bootstrapping tablet: real 0.021s user 0.020s sys 0.000s
I20240501 14:16:04.320578 15685 raft_consensus.cc:348] T acd35bd40253406fa1baa3f0d19c3d0a P 50161459714942cb9b4ae7743f76d07d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: true } }
I20240501 14:16:04.320667 15685 raft_consensus.cc:729] T acd35bd40253406fa1baa3f0d19c3d0a P 50161459714942cb9b4ae7743f76d07d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 50161459714942cb9b4ae7743f76d07d, State: Initialized, Role: LEARNER
I20240501 14:16:04.320755 15685 consensus_queue.cc:260] T acd35bd40253406fa1baa3f0d19c3d0a P 50161459714942cb9b4ae7743f76d07d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: true } }
I20240501 14:16:04.320961 15685 ts_tablet_manager.cc:1398] T acd35bd40253406fa1baa3f0d19c3d0a P 50161459714942cb9b4ae7743f76d07d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:16:04.321415 14306 tablet_copy_service.cc:342] P c87cbbf2d3ab4143815ef2c74ae760db: Request end of tablet copy session 50161459714942cb9b4ae7743f76d07d-acd35bd40253406fa1baa3f0d19c3d0a received from {username='slave'} at 127.7.39.133:55727
I20240501 14:16:04.321498 14306 tablet_copy_service.cc:433] P c87cbbf2d3ab4143815ef2c74ae760db: ending tablet copy session 50161459714942cb9b4ae7743f76d07d-acd35bd40253406fa1baa3f0d19c3d0a on tablet acd35bd40253406fa1baa3f0d19c3d0a with peer 50161459714942cb9b4ae7743f76d07d
I20240501 14:16:04.591395 15168 raft_consensus.cc:1203] T ca266243bcb84e8c9b65d1b096b5725d P 50161459714942cb9b4ae7743f76d07d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:16:04.704388 15299 raft_consensus.cc:1203] T 5f2b92ee1a1c46bcb36bc24563886d18 P e3770e9581534f77a7506253b8d201b0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:16:04.731356 15168 raft_consensus.cc:1203] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 50161459714942cb9b4ae7743f76d07d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:16:04.740824 15299 raft_consensus.cc:1203] T 00618ea639cf49d09cc3523b47cd0c07 P e3770e9581534f77a7506253b8d201b0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:16:04.785136 15299 raft_consensus.cc:1203] T 8ebd7203c0914f6c8e26ab4d20074765 P e3770e9581534f77a7506253b8d201b0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:16:04.785851 15168 raft_consensus.cc:1203] T acd35bd40253406fa1baa3f0d19c3d0a P 50161459714942cb9b4ae7743f76d07d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:16:04.818313 15168 raft_consensus.cc:1203] T 40732c47173c429aa69c6a41475849de P 50161459714942cb9b4ae7743f76d07d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:16:05.131006 15734 raft_consensus.cc:1050] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6: attempting to promote NON_VOTER 50161459714942cb9b4ae7743f76d07d to VOTER
I20240501 14:16:05.131567 15734 consensus_queue.cc:237] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } }
I20240501 14:16:05.133152 14417 raft_consensus.cc:1261] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Refusing update from remote peer e625ab1b03d14e2aae194052d2c67bc6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:16:05.133237 15168 raft_consensus.cc:1261] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 50161459714942cb9b4ae7743f76d07d [term 1 LEARNER]: Refusing update from remote peer e625ab1b03d14e2aae194052d2c67bc6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:16:05.133168 14286 raft_consensus.cc:1261] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Refusing update from remote peer e625ab1b03d14e2aae194052d2c67bc6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:16:05.133389 15517 raft_consensus.cc:1050] T ca266243bcb84e8c9b65d1b096b5725d P c87cbbf2d3ab4143815ef2c74ae760db: attempting to promote NON_VOTER 50161459714942cb9b4ae7743f76d07d to VOTER
I20240501 14:16:05.133641 15807 consensus_queue.cc:1035] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:16:05.133811 15493 consensus_queue.cc:1035] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:16:05.133921 15517 consensus_queue.cc:237] T ca266243bcb84e8c9b65d1b096b5725d P c87cbbf2d3ab4143815ef2c74ae760db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } }
I20240501 14:16:05.134202 15735 consensus_queue.cc:1035] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:16:05.135424 14418 raft_consensus.cc:1261] T ca266243bcb84e8c9b65d1b096b5725d P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Refusing update from remote peer c87cbbf2d3ab4143815ef2c74ae760db: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:16:05.135866 15809 consensus_queue.cc:1035] T ca266243bcb84e8c9b65d1b096b5725d P c87cbbf2d3ab4143815ef2c74ae760db [LEADER]: Connected to new peer: Peer: permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:16:05.136163 14548 raft_consensus.cc:1261] T ca266243bcb84e8c9b65d1b096b5725d P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Refusing update from remote peer c87cbbf2d3ab4143815ef2c74ae760db: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:16:05.136160 15807 raft_consensus.cc:2932] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 50161459714942cb9b4ae7743f76d07d (127.7.39.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } }
I20240501 14:16:05.136433 15167 raft_consensus.cc:1261] T ca266243bcb84e8c9b65d1b096b5725d P 50161459714942cb9b4ae7743f76d07d [term 1 LEARNER]: Refusing update from remote peer c87cbbf2d3ab4143815ef2c74ae760db: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:16:05.136562 14286 raft_consensus.cc:2932] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 50161459714942cb9b4ae7743f76d07d (127.7.39.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } }
I20240501 14:16:05.136889 15809 consensus_queue.cc:1035] T ca266243bcb84e8c9b65d1b096b5725d P c87cbbf2d3ab4143815ef2c74ae760db [LEADER]: Connected to new peer: Peer: permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:16:05.138419 14418 raft_consensus.cc:2932] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 50161459714942cb9b4ae7743f76d07d (127.7.39.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } }
I20240501 14:16:05.139127 15809 consensus_queue.cc:1035] T ca266243bcb84e8c9b65d1b096b5725d P c87cbbf2d3ab4143815ef2c74ae760db [LEADER]: Connected to new peer: Peer: permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:16:05.138938 14136 catalog_manager.cc:5484] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6 reported cstate change: config changed from index 4 to 5, 50161459714942cb9b4ae7743f76d07d (127.7.39.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e625ab1b03d14e2aae194052d2c67bc6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:16:05.140087 15168 raft_consensus.cc:2932] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 50161459714942cb9b4ae7743f76d07d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 50161459714942cb9b4ae7743f76d07d (127.7.39.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } }
I20240501 14:16:05.140797 15516 raft_consensus.cc:2932] T ca266243bcb84e8c9b65d1b096b5725d P c87cbbf2d3ab4143815ef2c74ae760db [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 50161459714942cb9b4ae7743f76d07d (127.7.39.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } }
I20240501 14:16:05.141222 14417 raft_consensus.cc:2932] T ca266243bcb84e8c9b65d1b096b5725d P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 50161459714942cb9b4ae7743f76d07d (127.7.39.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } }
I20240501 14:16:05.142113 14548 raft_consensus.cc:2932] T ca266243bcb84e8c9b65d1b096b5725d P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 50161459714942cb9b4ae7743f76d07d (127.7.39.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } }
I20240501 14:16:05.142721 15167 raft_consensus.cc:2932] T ca266243bcb84e8c9b65d1b096b5725d P 50161459714942cb9b4ae7743f76d07d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 50161459714942cb9b4ae7743f76d07d (127.7.39.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } }
I20240501 14:16:05.144481 14141 catalog_manager.cc:5484] T ca266243bcb84e8c9b65d1b096b5725d P 17062e38120042aaaca05e153b2c6ccf reported cstate change: config changed from index 4 to 5, 50161459714942cb9b4ae7743f76d07d (127.7.39.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } }
I20240501 14:16:05.148244 14286 consensus_queue.cc:237] T ca266243bcb84e8c9b65d1b096b5725d P c87cbbf2d3ab4143815ef2c74ae760db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } }
I20240501 14:16:05.149339 15168 raft_consensus.cc:1261] T ca266243bcb84e8c9b65d1b096b5725d P 50161459714942cb9b4ae7743f76d07d [term 1 FOLLOWER]: Refusing update from remote peer c87cbbf2d3ab4143815ef2c74ae760db: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:16:05.149339 14417 raft_consensus.cc:1261] T ca266243bcb84e8c9b65d1b096b5725d P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Refusing update from remote peer c87cbbf2d3ab4143815ef2c74ae760db: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:16:05.149703 15731 consensus_queue.cc:1035] T ca266243bcb84e8c9b65d1b096b5725d P c87cbbf2d3ab4143815ef2c74ae760db [LEADER]: Connected to new peer: Peer: permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:16:05.149844 15809 consensus_queue.cc:1035] T ca266243bcb84e8c9b65d1b096b5725d P c87cbbf2d3ab4143815ef2c74ae760db [LEADER]: Connected to new peer: Peer: permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:16:05.151190 15809 raft_consensus.cc:2932] T ca266243bcb84e8c9b65d1b096b5725d P c87cbbf2d3ab4143815ef2c74ae760db [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } }
I20240501 14:16:05.151659 15168 raft_consensus.cc:2932] T ca266243bcb84e8c9b65d1b096b5725d P 50161459714942cb9b4ae7743f76d07d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } }
I20240501 14:16:05.151890 14417 raft_consensus.cc:2932] T ca266243bcb84e8c9b65d1b096b5725d P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } }
I20240501 14:16:05.153987 14136 catalog_manager.cc:5484] T ca266243bcb84e8c9b65d1b096b5725d P 50161459714942cb9b4ae7743f76d07d reported cstate change: config changed from index 5 to 6, VOTER e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131) evicted. New cstate: current_term: 1 leader_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } }
I20240501 14:16:05.154165 14128 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ca266243bcb84e8c9b65d1b096b5725d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:16:05.155321 14548 consensus_queue.cc:237] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } }
I20240501 14:16:05.156827 14528 tablet_service.cc:1514] Processing DeleteTablet for tablet ca266243bcb84e8c9b65d1b096b5725d with delete_type TABLET_DATA_TOMBSTONED (TS e625ab1b03d14e2aae194052d2c67bc6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46722
I20240501 14:16:05.157042 14286 raft_consensus.cc:1261] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Refusing update from remote peer e625ab1b03d14e2aae194052d2c67bc6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:16:05.157470 15168 raft_consensus.cc:1261] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 50161459714942cb9b4ae7743f76d07d [term 1 FOLLOWER]: Refusing update from remote peer e625ab1b03d14e2aae194052d2c67bc6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:16:05.157438 15734 consensus_queue.cc:1035] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:16:05.157855 15734 consensus_queue.cc:1035] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:16:05.158834 15821 tablet_replica.cc:330] T ca266243bcb84e8c9b65d1b096b5725d P e625ab1b03d14e2aae194052d2c67bc6: stopping tablet replica
I20240501 14:16:05.158969 15734 raft_consensus.cc:2932] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 17062e38120042aaaca05e153b2c6ccf (127.7.39.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } }
I20240501 14:16:05.159196 15821 raft_consensus.cc:2229] T ca266243bcb84e8c9b65d1b096b5725d P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:16:05.159389 15821 raft_consensus.cc:2258] T ca266243bcb84e8c9b65d1b096b5725d P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:16:05.160537 15821 ts_tablet_manager.cc:1875] T ca266243bcb84e8c9b65d1b096b5725d P e625ab1b03d14e2aae194052d2c67bc6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:16:05.160547 14131 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 47ef7ee4ea3b46cbbac2126ceaa5fa24 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:16:05.160833 14285 raft_consensus.cc:2932] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 17062e38120042aaaca05e153b2c6ccf (127.7.39.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } }
I20240501 14:16:05.161382 14139 catalog_manager.cc:5484] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6 reported cstate change: config changed from index 5 to 6, VOTER 17062e38120042aaaca05e153b2c6ccf (127.7.39.130) evicted. New cstate: current_term: 1 leader_uuid: "e625ab1b03d14e2aae194052d2c67bc6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:16:05.162823 15821 ts_tablet_manager.cc:1888] T ca266243bcb84e8c9b65d1b096b5725d P e625ab1b03d14e2aae194052d2c67bc6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:16:05.162935 15821 log.cc:1196] T ca266243bcb84e8c9b65d1b096b5725d P e625ab1b03d14e2aae194052d2c67bc6: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal/wals/ca266243bcb84e8c9b65d1b096b5725d
I20240501 14:16:05.163352 15168 raft_consensus.cc:2932] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 50161459714942cb9b4ae7743f76d07d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 17062e38120042aaaca05e153b2c6ccf (127.7.39.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } }
I20240501 14:16:05.163581 14131 catalog_manager.cc:4830] TS e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131:33199): tablet ca266243bcb84e8c9b65d1b096b5725d (table rebalance_test_table_1 [id=2938421fb092484186ed564e78a8fd87]) successfully deleted
I20240501 14:16:05.164098 14397 tablet_service.cc:1514] Processing DeleteTablet for tablet 47ef7ee4ea3b46cbbac2126ceaa5fa24 with delete_type TABLET_DATA_TOMBSTONED (TS 17062e38120042aaaca05e153b2c6ccf not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36870
I20240501 14:16:05.164626 15822 tablet_replica.cc:330] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 17062e38120042aaaca05e153b2c6ccf: stopping tablet replica
I20240501 14:16:05.164994 15822 raft_consensus.cc:2229] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:16:05.165431 15822 raft_consensus.cc:2258] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:16:05.166565 15822 ts_tablet_manager.cc:1875] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 17062e38120042aaaca05e153b2c6ccf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:16:05.168350 15822 ts_tablet_manager.cc:1888] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 17062e38120042aaaca05e153b2c6ccf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:16:05.168438 15822 log.cc:1196] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 17062e38120042aaaca05e153b2c6ccf: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal/wals/47ef7ee4ea3b46cbbac2126ceaa5fa24
I20240501 14:16:05.168988 14129 catalog_manager.cc:4830] TS 17062e38120042aaaca05e153b2c6ccf (127.7.39.130:43603): tablet 47ef7ee4ea3b46cbbac2126ceaa5fa24 (table rebalance_test_table_2 [id=1176c470113c40c796085916c4a929d7]) successfully deleted
I20240501 14:16:05.183794 15732 raft_consensus.cc:1050] T 00618ea639cf49d09cc3523b47cd0c07 P 17062e38120042aaaca05e153b2c6ccf: attempting to promote NON_VOTER e3770e9581534f77a7506253b8d201b0 to VOTER
I20240501 14:16:05.184201 15732 consensus_queue.cc:237] T 00618ea639cf49d09cc3523b47cd0c07 P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } }
I20240501 14:16:05.185497 14548 raft_consensus.cc:1261] T 00618ea639cf49d09cc3523b47cd0c07 P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Refusing update from remote peer 17062e38120042aaaca05e153b2c6ccf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:16:05.185580 15299 raft_consensus.cc:1261] T 00618ea639cf49d09cc3523b47cd0c07 P e3770e9581534f77a7506253b8d201b0 [term 1 LEARNER]: Refusing update from remote peer 17062e38120042aaaca05e153b2c6ccf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:16:05.185580 14285 raft_consensus.cc:1261] T 00618ea639cf49d09cc3523b47cd0c07 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Refusing update from remote peer 17062e38120042aaaca05e153b2c6ccf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:16:05.185921 15732 consensus_queue.cc:1035] T 00618ea639cf49d09cc3523b47cd0c07 P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:16:05.186125 15824 consensus_queue.cc:1035] T 00618ea639cf49d09cc3523b47cd0c07 P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:16:05.186367 15506 consensus_queue.cc:1035] T 00618ea639cf49d09cc3523b47cd0c07 P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:16:05.188771 15732 raft_consensus.cc:2932] T 00618ea639cf49d09cc3523b47cd0c07 P 17062e38120042aaaca05e153b2c6ccf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e3770e9581534f77a7506253b8d201b0 (127.7.39.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } }
I20240501 14:16:05.189239 15299 raft_consensus.cc:2932] T 00618ea639cf49d09cc3523b47cd0c07 P e3770e9581534f77a7506253b8d201b0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e3770e9581534f77a7506253b8d201b0 (127.7.39.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } }
I20240501 14:16:05.190451 14548 raft_consensus.cc:2932] T 00618ea639cf49d09cc3523b47cd0c07 P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e3770e9581534f77a7506253b8d201b0 (127.7.39.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } }
I20240501 14:16:05.191507 14285 raft_consensus.cc:2932] T 00618ea639cf49d09cc3523b47cd0c07 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e3770e9581534f77a7506253b8d201b0 (127.7.39.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } }
I20240501 14:16:05.192688 14139 catalog_manager.cc:5484] T 00618ea639cf49d09cc3523b47cd0c07 P 17062e38120042aaaca05e153b2c6ccf reported cstate change: config changed from index 4 to 5, e3770e9581534f77a7506253b8d201b0 (127.7.39.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "17062e38120042aaaca05e153b2c6ccf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:16:05.196591 14417 consensus_queue.cc:237] T 00618ea639cf49d09cc3523b47cd0c07 P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } }
I20240501 14:16:05.197535 15299 raft_consensus.cc:1261] T 00618ea639cf49d09cc3523b47cd0c07 P e3770e9581534f77a7506253b8d201b0 [term 1 FOLLOWER]: Refusing update from remote peer 17062e38120042aaaca05e153b2c6ccf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:16:05.197571 14285 raft_consensus.cc:1261] T 00618ea639cf49d09cc3523b47cd0c07 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Refusing update from remote peer 17062e38120042aaaca05e153b2c6ccf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:16:05.197865 15824 consensus_queue.cc:1035] T 00618ea639cf49d09cc3523b47cd0c07 P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:16:05.198032 15732 consensus_queue.cc:1035] T 00618ea639cf49d09cc3523b47cd0c07 P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:16:05.199571 15733 raft_consensus.cc:2932] T 00618ea639cf49d09cc3523b47cd0c07 P 17062e38120042aaaca05e153b2c6ccf [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } }
I20240501 14:16:05.200074 14285 raft_consensus.cc:2932] T 00618ea639cf49d09cc3523b47cd0c07 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } }
I20240501 14:16:05.200084 15299 raft_consensus.cc:2932] T 00618ea639cf49d09cc3523b47cd0c07 P e3770e9581534f77a7506253b8d201b0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } }
I20240501 14:16:05.201099 14129 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 00618ea639cf49d09cc3523b47cd0c07 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:16:05.202258 14141 catalog_manager.cc:5484] T 00618ea639cf49d09cc3523b47cd0c07 P 17062e38120042aaaca05e153b2c6ccf reported cstate change: config changed from index 5 to 6, VOTER e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131) evicted. New cstate: current_term: 1 leader_uuid: "17062e38120042aaaca05e153b2c6ccf" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:16:05.204564 14528 tablet_service.cc:1514] Processing DeleteTablet for tablet 00618ea639cf49d09cc3523b47cd0c07 with delete_type TABLET_DATA_TOMBSTONED (TS e625ab1b03d14e2aae194052d2c67bc6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46722
I20240501 14:16:05.204775 15821 tablet_replica.cc:330] T 00618ea639cf49d09cc3523b47cd0c07 P e625ab1b03d14e2aae194052d2c67bc6: stopping tablet replica
I20240501 14:16:05.204924 15821 raft_consensus.cc:2229] T 00618ea639cf49d09cc3523b47cd0c07 P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:16:05.205081 15821 raft_consensus.cc:2258] T 00618ea639cf49d09cc3523b47cd0c07 P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:16:05.205838 15821 ts_tablet_manager.cc:1875] T 00618ea639cf49d09cc3523b47cd0c07 P e625ab1b03d14e2aae194052d2c67bc6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:16:05.207499 15821 ts_tablet_manager.cc:1888] T 00618ea639cf49d09cc3523b47cd0c07 P e625ab1b03d14e2aae194052d2c67bc6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:16:05.207569 15821 log.cc:1196] T 00618ea639cf49d09cc3523b47cd0c07 P e625ab1b03d14e2aae194052d2c67bc6: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal/wals/00618ea639cf49d09cc3523b47cd0c07
I20240501 14:16:05.208019 14131 catalog_manager.cc:4830] TS e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131:33199): tablet 00618ea639cf49d09cc3523b47cd0c07 (table rebalance_test_table_2 [id=1176c470113c40c796085916c4a929d7]) successfully deleted
I20240501 14:16:05.259608 15824 raft_consensus.cc:1050] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf: attempting to promote NON_VOTER 50161459714942cb9b4ae7743f76d07d to VOTER
I20240501 14:16:05.260051 15824 consensus_queue.cc:237] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } }
I20240501 14:16:05.261432 15168 raft_consensus.cc:1261] T 40732c47173c429aa69c6a41475849de P 50161459714942cb9b4ae7743f76d07d [term 1 LEARNER]: Refusing update from remote peer 17062e38120042aaaca05e153b2c6ccf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:16:05.261771 14548 raft_consensus.cc:1261] T 40732c47173c429aa69c6a41475849de P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Refusing update from remote peer 17062e38120042aaaca05e153b2c6ccf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:16:05.261924 14285 raft_consensus.cc:1261] T 40732c47173c429aa69c6a41475849de P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Refusing update from remote peer 17062e38120042aaaca05e153b2c6ccf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:16:05.261874 15506 consensus_queue.cc:1035] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:16:05.262248 15506 consensus_queue.cc:1035] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:16:05.262452 15733 consensus_queue.cc:1035] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:16:05.264767 15733 raft_consensus.cc:2932] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 50161459714942cb9b4ae7743f76d07d (127.7.39.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } }
I20240501 14:16:05.265324 14548 raft_consensus.cc:2932] T 40732c47173c429aa69c6a41475849de P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 50161459714942cb9b4ae7743f76d07d (127.7.39.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } }
I20240501 14:16:05.265519 14285 raft_consensus.cc:2932] T 40732c47173c429aa69c6a41475849de P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 50161459714942cb9b4ae7743f76d07d (127.7.39.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } }
I20240501 14:16:05.267194 15168 raft_consensus.cc:2932] T 40732c47173c429aa69c6a41475849de P 50161459714942cb9b4ae7743f76d07d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 50161459714942cb9b4ae7743f76d07d (127.7.39.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } }
I20240501 14:16:05.267594 14138 catalog_manager.cc:5484] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf reported cstate change: config changed from index 4 to 5, 50161459714942cb9b4ae7743f76d07d (127.7.39.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "17062e38120042aaaca05e153b2c6ccf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:16:05.271155 14417 consensus_queue.cc:237] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } }
I20240501 14:16:05.272297 15168 raft_consensus.cc:1261] T 40732c47173c429aa69c6a41475849de P 50161459714942cb9b4ae7743f76d07d [term 1 FOLLOWER]: Refusing update from remote peer 17062e38120042aaaca05e153b2c6ccf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:16:05.272399 14285 raft_consensus.cc:1261] T 40732c47173c429aa69c6a41475849de P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Refusing update from remote peer 17062e38120042aaaca05e153b2c6ccf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:16:05.272819 15506 consensus_queue.cc:1035] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:16:05.273130 15506 consensus_queue.cc:1035] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:16:05.274580 15506 raft_consensus.cc:2932] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } }
I20240501 14:16:05.275277 15168 raft_consensus.cc:2932] T 40732c47173c429aa69c6a41475849de P 50161459714942cb9b4ae7743f76d07d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } }
I20240501 14:16:05.275281 14285 raft_consensus.cc:2932] T 40732c47173c429aa69c6a41475849de P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } }
I20240501 14:16:05.276724 14129 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 40732c47173c429aa69c6a41475849de with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:16:05.277225 14141 catalog_manager.cc:5484] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf reported cstate change: config changed from index 5 to 6, VOTER e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131) evicted. New cstate: current_term: 1 leader_uuid: "17062e38120042aaaca05e153b2c6ccf" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:16:05.280475 14528 tablet_service.cc:1514] Processing DeleteTablet for tablet 40732c47173c429aa69c6a41475849de with delete_type TABLET_DATA_TOMBSTONED (TS e625ab1b03d14e2aae194052d2c67bc6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46722
I20240501 14:16:05.280690 15821 tablet_replica.cc:330] T 40732c47173c429aa69c6a41475849de P e625ab1b03d14e2aae194052d2c67bc6: stopping tablet replica
I20240501 14:16:05.280697 15824 raft_consensus.cc:1050] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf: attempting to promote NON_VOTER e3770e9581534f77a7506253b8d201b0 to VOTER
I20240501 14:16:05.280823 15821 raft_consensus.cc:2229] T 40732c47173c429aa69c6a41475849de P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:16:05.280994 15821 raft_consensus.cc:2258] T 40732c47173c429aa69c6a41475849de P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:16:05.281080 15824 consensus_queue.cc:237] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } }
I20240501 14:16:05.281790 15821 ts_tablet_manager.cc:1875] T 40732c47173c429aa69c6a41475849de P e625ab1b03d14e2aae194052d2c67bc6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:16:05.282832 14548 raft_consensus.cc:1261] T 5f2b92ee1a1c46bcb36bc24563886d18 P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Refusing update from remote peer 17062e38120042aaaca05e153b2c6ccf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:16:05.283263 15299 raft_consensus.cc:1261] T 5f2b92ee1a1c46bcb36bc24563886d18 P e3770e9581534f77a7506253b8d201b0 [term 1 LEARNER]: Refusing update from remote peer 17062e38120042aaaca05e153b2c6ccf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:16:05.283385 15732 consensus_queue.cc:1035] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:16:05.284159 15821 ts_tablet_manager.cc:1888] T 40732c47173c429aa69c6a41475849de P e625ab1b03d14e2aae194052d2c67bc6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:16:05.284237 15821 log.cc:1196] T 40732c47173c429aa69c6a41475849de P e625ab1b03d14e2aae194052d2c67bc6: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal/wals/40732c47173c429aa69c6a41475849de
I20240501 14:16:05.284317 14285 raft_consensus.cc:1261] T 5f2b92ee1a1c46bcb36bc24563886d18 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Refusing update from remote peer 17062e38120042aaaca05e153b2c6ccf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:16:05.284783 15733 consensus_queue.cc:1035] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:16:05.285111 15493 raft_consensus.cc:1050] T 8ebd7203c0914f6c8e26ab4d20074765 P e625ab1b03d14e2aae194052d2c67bc6: attempting to promote NON_VOTER e3770e9581534f77a7506253b8d201b0 to VOTER
I20240501 14:16:05.285122 15733 consensus_queue.cc:1035] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:16:05.285473 14131 catalog_manager.cc:4830] TS e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131:33199): tablet 40732c47173c429aa69c6a41475849de (table rebalance_test_table_1 [id=2938421fb092484186ed564e78a8fd87]) successfully deleted
I20240501 14:16:05.285480 15493 consensus_queue.cc:237] T 8ebd7203c0914f6c8e26ab4d20074765 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } }
I20240501 14:16:05.286856 14286 raft_consensus.cc:1261] T 8ebd7203c0914f6c8e26ab4d20074765 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Refusing update from remote peer e625ab1b03d14e2aae194052d2c67bc6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:16:05.286996 14417 raft_consensus.cc:1261] T 8ebd7203c0914f6c8e26ab4d20074765 P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Refusing update from remote peer e625ab1b03d14e2aae194052d2c67bc6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:16:05.287346 15734 consensus_queue.cc:1035] T 8ebd7203c0914f6c8e26ab4d20074765 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:16:05.287360 15298 raft_consensus.cc:1261] T 8ebd7203c0914f6c8e26ab4d20074765 P e3770e9581534f77a7506253b8d201b0 [term 1 LEARNER]: Refusing update from remote peer e625ab1b03d14e2aae194052d2c67bc6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:16:05.287550 15807 consensus_queue.cc:1035] T 8ebd7203c0914f6c8e26ab4d20074765 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:16:05.287762 15735 consensus_queue.cc:1035] T 8ebd7203c0914f6c8e26ab4d20074765 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:16:05.290158 15807 raft_consensus.cc:2932] T 8ebd7203c0914f6c8e26ab4d20074765 P e625ab1b03d14e2aae194052d2c67bc6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e3770e9581534f77a7506253b8d201b0 (127.7.39.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } }
I20240501 14:16:05.290949 15733 raft_consensus.cc:2932] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e3770e9581534f77a7506253b8d201b0 (127.7.39.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } }
I20240501 14:16:05.291090 14285 raft_consensus.cc:2932] T 8ebd7203c0914f6c8e26ab4d20074765 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e3770e9581534f77a7506253b8d201b0 (127.7.39.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } }
I20240501 14:16:05.291148 15298 raft_consensus.cc:2932] T 8ebd7203c0914f6c8e26ab4d20074765 P e3770e9581534f77a7506253b8d201b0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e3770e9581534f77a7506253b8d201b0 (127.7.39.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } }
I20240501 14:16:05.293355 14548 raft_consensus.cc:2932] T 5f2b92ee1a1c46bcb36bc24563886d18 P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e3770e9581534f77a7506253b8d201b0 (127.7.39.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } }
I20240501 14:16:05.293434 14418 raft_consensus.cc:2932] T 8ebd7203c0914f6c8e26ab4d20074765 P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e3770e9581534f77a7506253b8d201b0 (127.7.39.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } }
I20240501 14:16:05.293463 14136 catalog_manager.cc:5484] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf reported cstate change: config changed from index 4 to 5, e3770e9581534f77a7506253b8d201b0 (127.7.39.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "17062e38120042aaaca05e153b2c6ccf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:16:05.295166 14285 raft_consensus.cc:2932] T 5f2b92ee1a1c46bcb36bc24563886d18 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e3770e9581534f77a7506253b8d201b0 (127.7.39.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } }
I20240501 14:16:05.295336 15299 raft_consensus.cc:2932] T 5f2b92ee1a1c46bcb36bc24563886d18 P e3770e9581534f77a7506253b8d201b0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e3770e9581534f77a7506253b8d201b0 (127.7.39.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } }
I20240501 14:16:05.295776 14141 catalog_manager.cc:5484] T 8ebd7203c0914f6c8e26ab4d20074765 P c87cbbf2d3ab4143815ef2c74ae760db reported cstate change: config changed from index 4 to 5, e3770e9581534f77a7506253b8d201b0 (127.7.39.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e625ab1b03d14e2aae194052d2c67bc6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } }
I20240501 14:16:05.298863 14548 consensus_queue.cc:237] T 8ebd7203c0914f6c8e26ab4d20074765 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } }
I20240501 14:16:05.300225 14285 raft_consensus.cc:1261] T 8ebd7203c0914f6c8e26ab4d20074765 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Refusing update from remote peer e625ab1b03d14e2aae194052d2c67bc6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:16:05.300225 15299 raft_consensus.cc:1261] T 8ebd7203c0914f6c8e26ab4d20074765 P e3770e9581534f77a7506253b8d201b0 [term 1 FOLLOWER]: Refusing update from remote peer e625ab1b03d14e2aae194052d2c67bc6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:16:05.300627 15734 consensus_queue.cc:1035] T 8ebd7203c0914f6c8e26ab4d20074765 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:16:05.300951 15734 consensus_queue.cc:1035] T 8ebd7203c0914f6c8e26ab4d20074765 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:16:05.301347 14418 consensus_queue.cc:237] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } }
I20240501 14:16:05.302541 15298 raft_consensus.cc:1261] T 5f2b92ee1a1c46bcb36bc24563886d18 P e3770e9581534f77a7506253b8d201b0 [term 1 FOLLOWER]: Refusing update from remote peer 17062e38120042aaaca05e153b2c6ccf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:16:05.302685 14548 raft_consensus.cc:1261] T 5f2b92ee1a1c46bcb36bc24563886d18 P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Refusing update from remote peer 17062e38120042aaaca05e153b2c6ccf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:16:05.302877 15732 consensus_queue.cc:1035] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:16:05.303092 15493 raft_consensus.cc:2932] T 8ebd7203c0914f6c8e26ab4d20074765 P e625ab1b03d14e2aae194052d2c67bc6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 17062e38120042aaaca05e153b2c6ccf (127.7.39.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } }
I20240501 14:16:05.303632 15732 consensus_queue.cc:1035] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:16:05.304819 15299 raft_consensus.cc:2932] T 8ebd7203c0914f6c8e26ab4d20074765 P e3770e9581534f77a7506253b8d201b0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 17062e38120042aaaca05e153b2c6ccf (127.7.39.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } }
I20240501 14:16:05.305218 14141 catalog_manager.cc:5484] T 8ebd7203c0914f6c8e26ab4d20074765 P e625ab1b03d14e2aae194052d2c67bc6 reported cstate change: config changed from index 5 to 6, VOTER 17062e38120042aaaca05e153b2c6ccf (127.7.39.130) evicted. New cstate: current_term: 1 leader_uuid: "e625ab1b03d14e2aae194052d2c67bc6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:16:05.305333 14286 raft_consensus.cc:2932] T 8ebd7203c0914f6c8e26ab4d20074765 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 17062e38120042aaaca05e153b2c6ccf (127.7.39.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } }
I20240501 14:16:05.307363 14131 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8ebd7203c0914f6c8e26ab4d20074765 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:16:05.307943 15732 raft_consensus.cc:2932] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } }
I20240501 14:16:05.308355 14548 raft_consensus.cc:2932] T 5f2b92ee1a1c46bcb36bc24563886d18 P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } }
I20240501 14:16:05.309448 14129 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5f2b92ee1a1c46bcb36bc24563886d18 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:16:05.309624 14397 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ebd7203c0914f6c8e26ab4d20074765 with delete_type TABLET_DATA_TOMBSTONED (TS 17062e38120042aaaca05e153b2c6ccf not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36870
I20240501 14:16:05.309804 15822 tablet_replica.cc:330] T 8ebd7203c0914f6c8e26ab4d20074765 P 17062e38120042aaaca05e153b2c6ccf: stopping tablet replica
I20240501 14:16:05.309945 15822 raft_consensus.cc:2229] T 8ebd7203c0914f6c8e26ab4d20074765 P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:16:05.310104 15822 raft_consensus.cc:2258] T 8ebd7203c0914f6c8e26ab4d20074765 P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:16:05.310662 15298 raft_consensus.cc:2932] T 5f2b92ee1a1c46bcb36bc24563886d18 P e3770e9581534f77a7506253b8d201b0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } }
I20240501 14:16:05.310825 14138 catalog_manager.cc:5484] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf reported cstate change: config changed from index 5 to 6, VOTER c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129) evicted. New cstate: current_term: 1 leader_uuid: "17062e38120042aaaca05e153b2c6ccf" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:16:05.313611 14264 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f2b92ee1a1c46bcb36bc24563886d18 with delete_type TABLET_DATA_TOMBSTONED (TS c87cbbf2d3ab4143815ef2c74ae760db not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51888
I20240501 14:16:05.313694 15822 ts_tablet_manager.cc:1875] T 8ebd7203c0914f6c8e26ab4d20074765 P 17062e38120042aaaca05e153b2c6ccf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:16:05.314152 15900 tablet_replica.cc:330] T 5f2b92ee1a1c46bcb36bc24563886d18 P c87cbbf2d3ab4143815ef2c74ae760db: stopping tablet replica
I20240501 14:16:05.314455 15900 raft_consensus.cc:2229] T 5f2b92ee1a1c46bcb36bc24563886d18 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:16:05.314695 15900 raft_consensus.cc:2258] T 5f2b92ee1a1c46bcb36bc24563886d18 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:16:05.315773 15822 ts_tablet_manager.cc:1888] T 8ebd7203c0914f6c8e26ab4d20074765 P 17062e38120042aaaca05e153b2c6ccf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:16:05.315848 15822 log.cc:1196] T 8ebd7203c0914f6c8e26ab4d20074765 P 17062e38120042aaaca05e153b2c6ccf: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal/wals/8ebd7203c0914f6c8e26ab4d20074765
I20240501 14:16:05.316197 15900 ts_tablet_manager.cc:1875] T 5f2b92ee1a1c46bcb36bc24563886d18 P c87cbbf2d3ab4143815ef2c74ae760db: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:16:05.316452 14129 catalog_manager.cc:4830] TS 17062e38120042aaaca05e153b2c6ccf (127.7.39.130:43603): tablet 8ebd7203c0914f6c8e26ab4d20074765 (table rebalance_test_table_1 [id=2938421fb092484186ed564e78a8fd87]) successfully deleted
I20240501 14:16:05.318372 15900 ts_tablet_manager.cc:1888] T 5f2b92ee1a1c46bcb36bc24563886d18 P c87cbbf2d3ab4143815ef2c74ae760db: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:16:05.318464 15900 log.cc:1196] T 5f2b92ee1a1c46bcb36bc24563886d18 P c87cbbf2d3ab4143815ef2c74ae760db: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal/wals/5f2b92ee1a1c46bcb36bc24563886d18
I20240501 14:16:05.319118 14128 catalog_manager.cc:4830] TS c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129:42369): tablet 5f2b92ee1a1c46bcb36bc24563886d18 (table rebalance_test_table_0 [id=066c15421d0b476bbe4204501666d610]) successfully deleted
I20240501 14:16:05.392262 15809 raft_consensus.cc:1050] T acd35bd40253406fa1baa3f0d19c3d0a P c87cbbf2d3ab4143815ef2c74ae760db: attempting to promote NON_VOTER 50161459714942cb9b4ae7743f76d07d to VOTER
I20240501 14:16:05.392707 15809 consensus_queue.cc:237] T acd35bd40253406fa1baa3f0d19c3d0a P c87cbbf2d3ab4143815ef2c74ae760db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } }
I20240501 14:16:05.394160 15168 raft_consensus.cc:1261] T acd35bd40253406fa1baa3f0d19c3d0a P 50161459714942cb9b4ae7743f76d07d [term 1 LEARNER]: Refusing update from remote peer c87cbbf2d3ab4143815ef2c74ae760db: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:16:05.394201 14418 raft_consensus.cc:1261] T acd35bd40253406fa1baa3f0d19c3d0a P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Refusing update from remote peer c87cbbf2d3ab4143815ef2c74ae760db: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:16:05.394272 14548 raft_consensus.cc:1261] T acd35bd40253406fa1baa3f0d19c3d0a P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Refusing update from remote peer c87cbbf2d3ab4143815ef2c74ae760db: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:16:05.394624 15517 consensus_queue.cc:1035] T acd35bd40253406fa1baa3f0d19c3d0a P c87cbbf2d3ab4143815ef2c74ae760db [LEADER]: Connected to new peer: Peer: permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:16:05.394855 15731 consensus_queue.cc:1035] T acd35bd40253406fa1baa3f0d19c3d0a P c87cbbf2d3ab4143815ef2c74ae760db [LEADER]: Connected to new peer: Peer: permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:16:05.395037 15809 consensus_queue.cc:1035] T acd35bd40253406fa1baa3f0d19c3d0a P c87cbbf2d3ab4143815ef2c74ae760db [LEADER]: Connected to new peer: Peer: permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:16:05.397071 15731 raft_consensus.cc:2932] T acd35bd40253406fa1baa3f0d19c3d0a P c87cbbf2d3ab4143815ef2c74ae760db [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 50161459714942cb9b4ae7743f76d07d (127.7.39.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } }
I20240501 14:16:05.397588 14418 raft_consensus.cc:2932] T acd35bd40253406fa1baa3f0d19c3d0a P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 50161459714942cb9b4ae7743f76d07d (127.7.39.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } }
I20240501 14:16:05.397588 15168 raft_consensus.cc:2932] T acd35bd40253406fa1baa3f0d19c3d0a P 50161459714942cb9b4ae7743f76d07d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 50161459714942cb9b4ae7743f76d07d (127.7.39.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } }
I20240501 14:16:05.399206 14548 raft_consensus.cc:2932] T acd35bd40253406fa1baa3f0d19c3d0a P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 50161459714942cb9b4ae7743f76d07d (127.7.39.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } }
I20240501 14:16:05.400264 14141 catalog_manager.cc:5484] T acd35bd40253406fa1baa3f0d19c3d0a P 50161459714942cb9b4ae7743f76d07d reported cstate change: config changed from index 4 to 5, 50161459714942cb9b4ae7743f76d07d (127.7.39.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } }
I20240501 14:16:05.403527 14286 consensus_queue.cc:237] T acd35bd40253406fa1baa3f0d19c3d0a P c87cbbf2d3ab4143815ef2c74ae760db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } }
I20240501 14:16:05.404534 15168 raft_consensus.cc:1261] T acd35bd40253406fa1baa3f0d19c3d0a P 50161459714942cb9b4ae7743f76d07d [term 1 FOLLOWER]: Refusing update from remote peer c87cbbf2d3ab4143815ef2c74ae760db: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:16:05.404646 14548 raft_consensus.cc:1261] T acd35bd40253406fa1baa3f0d19c3d0a P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Refusing update from remote peer c87cbbf2d3ab4143815ef2c74ae760db: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:16:05.404873 15809 consensus_queue.cc:1035] T acd35bd40253406fa1baa3f0d19c3d0a P c87cbbf2d3ab4143815ef2c74ae760db [LEADER]: Connected to new peer: Peer: permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:16:05.405031 15516 consensus_queue.cc:1035] T acd35bd40253406fa1baa3f0d19c3d0a P c87cbbf2d3ab4143815ef2c74ae760db [LEADER]: Connected to new peer: Peer: permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:16:05.406530 15517 raft_consensus.cc:2932] T acd35bd40253406fa1baa3f0d19c3d0a P c87cbbf2d3ab4143815ef2c74ae760db [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 17062e38120042aaaca05e153b2c6ccf (127.7.39.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } }
I20240501 14:16:05.406920 15168 raft_consensus.cc:2932] T acd35bd40253406fa1baa3f0d19c3d0a P 50161459714942cb9b4ae7743f76d07d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 17062e38120042aaaca05e153b2c6ccf (127.7.39.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } }
I20240501 14:16:05.408167 14128 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet acd35bd40253406fa1baa3f0d19c3d0a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:16:05.408646 14548 raft_consensus.cc:2932] T acd35bd40253406fa1baa3f0d19c3d0a P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 17062e38120042aaaca05e153b2c6ccf (127.7.39.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } }
I20240501 14:16:05.408874 14139 catalog_manager.cc:5484] T acd35bd40253406fa1baa3f0d19c3d0a P 50161459714942cb9b4ae7743f76d07d reported cstate change: config changed from index 5 to 6, VOTER 17062e38120042aaaca05e153b2c6ccf (127.7.39.130) evicted. New cstate: current_term: 1 leader_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } }
I20240501 14:16:05.411024 14397 tablet_service.cc:1514] Processing DeleteTablet for tablet acd35bd40253406fa1baa3f0d19c3d0a with delete_type TABLET_DATA_TOMBSTONED (TS 17062e38120042aaaca05e153b2c6ccf not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36870
I20240501 14:16:05.411285 15822 tablet_replica.cc:330] T acd35bd40253406fa1baa3f0d19c3d0a P 17062e38120042aaaca05e153b2c6ccf: stopping tablet replica
I20240501 14:16:05.411419 15822 raft_consensus.cc:2229] T acd35bd40253406fa1baa3f0d19c3d0a P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:16:05.411593 15822 raft_consensus.cc:2258] T acd35bd40253406fa1baa3f0d19c3d0a P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:16:05.412231 15822 ts_tablet_manager.cc:1875] T acd35bd40253406fa1baa3f0d19c3d0a P 17062e38120042aaaca05e153b2c6ccf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:16:05.413924 15822 ts_tablet_manager.cc:1888] T acd35bd40253406fa1baa3f0d19c3d0a P 17062e38120042aaaca05e153b2c6ccf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:16:05.414001 15822 log.cc:1196] T acd35bd40253406fa1baa3f0d19c3d0a P 17062e38120042aaaca05e153b2c6ccf: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal/wals/acd35bd40253406fa1baa3f0d19c3d0a
I20240501 14:16:05.414453 14129 catalog_manager.cc:4830] TS 17062e38120042aaaca05e153b2c6ccf (127.7.39.130:43603): tablet acd35bd40253406fa1baa3f0d19c3d0a (table rebalance_test_table_2 [id=1176c470113c40c796085916c4a929d7]) successfully deleted
I20240501 14:16:05.592025 14397 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:16:05.594472 15148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:16:05.594736 15279 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:16:05.595135 14528 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:16:05.607232 14264 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:16:05.638729 14286 consensus_queue.cc:237] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: NON_VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: true } }
I20240501 14:16:05.640168 14548 raft_consensus.cc:1261] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Refusing update from remote peer c87cbbf2d3ab4143815ef2c74ae760db: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:16:05.640429 14418 raft_consensus.cc:1261] T 3ded831efb0846a192400e8856dbd9b6 P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Refusing update from remote peer c87cbbf2d3ab4143815ef2c74ae760db: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:16:05.640573 15516 consensus_queue.cc:1035] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db [LEADER]: Connected to new peer: Peer: permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:16:05.640851 15809 consensus_queue.cc:1035] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db [LEADER]: Connected to new peer: Peer: permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:16:05.642433 15516 raft_consensus.cc:2932] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e3770e9581534f77a7506253b8d201b0 (127.7.39.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: NON_VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: true } } }
I20240501 14:16:05.642833 14548 raft_consensus.cc:2932] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e3770e9581534f77a7506253b8d201b0 (127.7.39.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: NON_VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: true } } }
I20240501 14:16:05.643424 14418 raft_consensus.cc:2932] T 3ded831efb0846a192400e8856dbd9b6 P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e3770e9581534f77a7506253b8d201b0 (127.7.39.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: NON_VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: true } } }
I20240501 14:16:05.646025 14138 catalog_manager.cc:5484] T 3ded831efb0846a192400e8856dbd9b6 P 17062e38120042aaaca05e153b2c6ccf reported cstate change: config changed from index -1 to 4, NON_VOTER e3770e9581534f77a7506253b8d201b0 (127.7.39.132) added. New cstate: current_term: 1 leader_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: NON_VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: true } } }
W20240501 14:16:05.649416 14219 consensus_peers.cc:485] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db -> Peer e3770e9581534f77a7506253b8d201b0 (127.7.39.132:38927): Couldn't send request to peer e3770e9581534f77a7506253b8d201b0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3ded831efb0846a192400e8856dbd9b6. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:16:05.653368 14286 consensus_queue.cc:237] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: true } }
I20240501 14:16:05.654544 14418 raft_consensus.cc:1261] T 84e2150d96c4451d95302894342228e8 P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Refusing update from remote peer c87cbbf2d3ab4143815ef2c74ae760db: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:16:05.654870 14221 consensus_peers.cc:485] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db -> Peer 50161459714942cb9b4ae7743f76d07d (127.7.39.133:36729): Couldn't send request to peer 50161459714942cb9b4ae7743f76d07d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 84e2150d96c4451d95302894342228e8. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:16:05.654878 14548 raft_consensus.cc:1261] T 84e2150d96c4451d95302894342228e8 P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Refusing update from remote peer c87cbbf2d3ab4143815ef2c74ae760db: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:16:05.655114 15517 consensus_queue.cc:1035] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db [LEADER]: Connected to new peer: Peer: permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:16:05.655277 15809 consensus_queue.cc:1035] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db [LEADER]: Connected to new peer: Peer: permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:16:05.657243 15731 raft_consensus.cc:2932] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 50161459714942cb9b4ae7743f76d07d (127.7.39.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: true } } }
I20240501 14:16:05.657516 14548 raft_consensus.cc:2932] T 84e2150d96c4451d95302894342228e8 P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 50161459714942cb9b4ae7743f76d07d (127.7.39.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: true } } }
I20240501 14:16:05.657709 14418 raft_consensus.cc:2932] T 84e2150d96c4451d95302894342228e8 P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 50161459714942cb9b4ae7743f76d07d (127.7.39.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: true } } }
I20240501 14:16:05.659765 14141 catalog_manager.cc:5484] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db reported cstate change: config changed from index -1 to 4, NON_VOTER 50161459714942cb9b4ae7743f76d07d (127.7.39.133) added. New cstate: current_term: 1 leader_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:16:06.040082 15992 ts_tablet_manager.cc:902] T 3ded831efb0846a192400e8856dbd9b6 P e3770e9581534f77a7506253b8d201b0: Initiating tablet copy from peer c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129:42369)
I20240501 14:16:06.040619 15992 tablet_copy_client.cc:308] T 3ded831efb0846a192400e8856dbd9b6 P e3770e9581534f77a7506253b8d201b0: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.129:42369
I20240501 14:16:06.044046 14306 tablet_copy_service.cc:140] P c87cbbf2d3ab4143815ef2c74ae760db: Received BeginTabletCopySession request for tablet 3ded831efb0846a192400e8856dbd9b6 from peer e3770e9581534f77a7506253b8d201b0 ({username='slave'} at 127.7.39.132:43117)
I20240501 14:16:06.044152 14306 tablet_copy_service.cc:161] P c87cbbf2d3ab4143815ef2c74ae760db: Beginning new tablet copy session on tablet 3ded831efb0846a192400e8856dbd9b6 from peer e3770e9581534f77a7506253b8d201b0 at {username='slave'} at 127.7.39.132:43117: session id = e3770e9581534f77a7506253b8d201b0-3ded831efb0846a192400e8856dbd9b6
I20240501 14:16:06.044888 14306 tablet_copy_source_session.cc:205] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:16:06.045773 15992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ded831efb0846a192400e8856dbd9b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:06.047652 15992 tablet_copy_client.cc:783] T 3ded831efb0846a192400e8856dbd9b6 P e3770e9581534f77a7506253b8d201b0: tablet copy: Starting download of 0 data blocks...
I20240501 14:16:06.047812 15992 tablet_copy_client.cc:647] T 3ded831efb0846a192400e8856dbd9b6 P e3770e9581534f77a7506253b8d201b0: tablet copy: Starting download of 1 WAL segments...
I20240501 14:16:06.048987 15992 tablet_copy_client.cc:523] T 3ded831efb0846a192400e8856dbd9b6 P e3770e9581534f77a7506253b8d201b0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:16:06.049901 15992 tablet_bootstrap.cc:492] T 3ded831efb0846a192400e8856dbd9b6 P e3770e9581534f77a7506253b8d201b0: Bootstrap starting.
I20240501 14:16:06.065052 15992 tablet_bootstrap.cc:492] T 3ded831efb0846a192400e8856dbd9b6 P e3770e9581534f77a7506253b8d201b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:16:06.065295 15992 tablet_bootstrap.cc:492] T 3ded831efb0846a192400e8856dbd9b6 P e3770e9581534f77a7506253b8d201b0: Bootstrap complete.
I20240501 14:16:06.065425 15992 ts_tablet_manager.cc:1367] T 3ded831efb0846a192400e8856dbd9b6 P e3770e9581534f77a7506253b8d201b0: Time spent bootstrapping tablet: real 0.016s user 0.014s sys 0.000s
I20240501 14:16:06.065775 15992 raft_consensus.cc:348] T 3ded831efb0846a192400e8856dbd9b6 P e3770e9581534f77a7506253b8d201b0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: NON_VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: true } }
I20240501 14:16:06.065874 15992 raft_consensus.cc:729] T 3ded831efb0846a192400e8856dbd9b6 P e3770e9581534f77a7506253b8d201b0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e3770e9581534f77a7506253b8d201b0, State: Initialized, Role: LEARNER
I20240501 14:16:06.065955 15992 consensus_queue.cc:260] T 3ded831efb0846a192400e8856dbd9b6 P e3770e9581534f77a7506253b8d201b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: NON_VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: true } }
I20240501 14:16:06.066313 15992 ts_tablet_manager.cc:1398] T 3ded831efb0846a192400e8856dbd9b6 P e3770e9581534f77a7506253b8d201b0: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240501 14:16:06.066952 14306 tablet_copy_service.cc:342] P c87cbbf2d3ab4143815ef2c74ae760db: Request end of tablet copy session e3770e9581534f77a7506253b8d201b0-3ded831efb0846a192400e8856dbd9b6 received from {username='slave'} at 127.7.39.132:43117
I20240501 14:16:06.067034 14306 tablet_copy_service.cc:433] P c87cbbf2d3ab4143815ef2c74ae760db: ending tablet copy session e3770e9581534f77a7506253b8d201b0-3ded831efb0846a192400e8856dbd9b6 on tablet 3ded831efb0846a192400e8856dbd9b6 with peer e3770e9581534f77a7506253b8d201b0
I20240501 14:16:06.154551 16010 ts_tablet_manager.cc:902] T 84e2150d96c4451d95302894342228e8 P 50161459714942cb9b4ae7743f76d07d: Initiating tablet copy from peer c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129:42369)
I20240501 14:16:06.155014 16010 tablet_copy_client.cc:308] T 84e2150d96c4451d95302894342228e8 P 50161459714942cb9b4ae7743f76d07d: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.129:42369
I20240501 14:16:06.155537 14306 tablet_copy_service.cc:140] P c87cbbf2d3ab4143815ef2c74ae760db: Received BeginTabletCopySession request for tablet 84e2150d96c4451d95302894342228e8 from peer 50161459714942cb9b4ae7743f76d07d ({username='slave'} at 127.7.39.133:55727)
I20240501 14:16:06.155659 14306 tablet_copy_service.cc:161] P c87cbbf2d3ab4143815ef2c74ae760db: Beginning new tablet copy session on tablet 84e2150d96c4451d95302894342228e8 from peer 50161459714942cb9b4ae7743f76d07d at {username='slave'} at 127.7.39.133:55727: session id = 50161459714942cb9b4ae7743f76d07d-84e2150d96c4451d95302894342228e8
I20240501 14:16:06.156607 14306 tablet_copy_source_session.cc:205] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:16:06.157470 16010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84e2150d96c4451d95302894342228e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:06.159471 16010 tablet_copy_client.cc:783] T 84e2150d96c4451d95302894342228e8 P 50161459714942cb9b4ae7743f76d07d: tablet copy: Starting download of 0 data blocks...
I20240501 14:16:06.159628 16010 tablet_copy_client.cc:647] T 84e2150d96c4451d95302894342228e8 P 50161459714942cb9b4ae7743f76d07d: tablet copy: Starting download of 1 WAL segments...
I20240501 14:16:06.160677 16010 tablet_copy_client.cc:523] T 84e2150d96c4451d95302894342228e8 P 50161459714942cb9b4ae7743f76d07d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:16:06.161574 16010 tablet_bootstrap.cc:492] T 84e2150d96c4451d95302894342228e8 P 50161459714942cb9b4ae7743f76d07d: Bootstrap starting.
I20240501 14:16:06.177742 16010 tablet_bootstrap.cc:492] T 84e2150d96c4451d95302894342228e8 P 50161459714942cb9b4ae7743f76d07d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:16:06.177984 16010 tablet_bootstrap.cc:492] T 84e2150d96c4451d95302894342228e8 P 50161459714942cb9b4ae7743f76d07d: Bootstrap complete.
I20240501 14:16:06.178135 16010 ts_tablet_manager.cc:1367] T 84e2150d96c4451d95302894342228e8 P 50161459714942cb9b4ae7743f76d07d: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.003s
I20240501 14:16:06.178488 16010 raft_consensus.cc:348] T 84e2150d96c4451d95302894342228e8 P 50161459714942cb9b4ae7743f76d07d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: true } }
I20240501 14:16:06.178587 16010 raft_consensus.cc:729] T 84e2150d96c4451d95302894342228e8 P 50161459714942cb9b4ae7743f76d07d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 50161459714942cb9b4ae7743f76d07d, State: Initialized, Role: LEARNER
I20240501 14:16:06.178675 16010 consensus_queue.cc:260] T 84e2150d96c4451d95302894342228e8 P 50161459714942cb9b4ae7743f76d07d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: NON_VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: true } }
I20240501 14:16:06.178974 16010 ts_tablet_manager.cc:1398] T 84e2150d96c4451d95302894342228e8 P 50161459714942cb9b4ae7743f76d07d: Time spent starting tablet: real 0.001s user 0.003s sys 0.001s
I20240501 14:16:06.179479 14306 tablet_copy_service.cc:342] P c87cbbf2d3ab4143815ef2c74ae760db: Request end of tablet copy session 50161459714942cb9b4ae7743f76d07d-84e2150d96c4451d95302894342228e8 received from {username='slave'} at 127.7.39.133:55727
I20240501 14:16:06.179562 14306 tablet_copy_service.cc:433] P c87cbbf2d3ab4143815ef2c74ae760db: ending tablet copy session 50161459714942cb9b4ae7743f76d07d-84e2150d96c4451d95302894342228e8 on tablet 84e2150d96c4451d95302894342228e8 with peer 50161459714942cb9b4ae7743f76d07d
I20240501 14:16:06.574744 15298 raft_consensus.cc:1203] T 3ded831efb0846a192400e8856dbd9b6 P e3770e9581534f77a7506253b8d201b0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:16:06.628119 15168 raft_consensus.cc:1203] T 84e2150d96c4451d95302894342228e8 P 50161459714942cb9b4ae7743f76d07d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:16:06.991221 16038 raft_consensus.cc:1050] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db: attempting to promote NON_VOTER e3770e9581534f77a7506253b8d201b0 to VOTER
I20240501 14:16:06.991576 16038 consensus_queue.cc:237] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } }
I20240501 14:16:06.992808 14548 raft_consensus.cc:1261] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Refusing update from remote peer c87cbbf2d3ab4143815ef2c74ae760db: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:16:06.992831 14418 raft_consensus.cc:1261] T 3ded831efb0846a192400e8856dbd9b6 P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Refusing update from remote peer c87cbbf2d3ab4143815ef2c74ae760db: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:16:06.992866 15298 raft_consensus.cc:1261] T 3ded831efb0846a192400e8856dbd9b6 P e3770e9581534f77a7506253b8d201b0 [term 1 LEARNER]: Refusing update from remote peer c87cbbf2d3ab4143815ef2c74ae760db: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:16:06.993263 16037 consensus_queue.cc:1035] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:16:06.993454 16061 consensus_queue.cc:1035] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db [LEADER]: Connected to new peer: Peer: permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:16:06.993629 15809 consensus_queue.cc:1035] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db [LEADER]: Connected to new peer: Peer: permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:16:06.995559 16037 raft_consensus.cc:2932] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e3770e9581534f77a7506253b8d201b0 (127.7.39.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } }
I20240501 14:16:06.995975 14548 raft_consensus.cc:2932] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e3770e9581534f77a7506253b8d201b0 (127.7.39.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } }
I20240501 14:16:06.996091 15298 raft_consensus.cc:2932] T 3ded831efb0846a192400e8856dbd9b6 P e3770e9581534f77a7506253b8d201b0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e3770e9581534f77a7506253b8d201b0 (127.7.39.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } }
I20240501 14:16:06.997951 14418 raft_consensus.cc:2932] T 3ded831efb0846a192400e8856dbd9b6 P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e3770e9581534f77a7506253b8d201b0 (127.7.39.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } }
I20240501 14:16:06.998593 14139 catalog_manager.cc:5484] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db reported cstate change: config changed from index 4 to 5, e3770e9581534f77a7506253b8d201b0 (127.7.39.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:16:07.007797 16061 raft_consensus.cc:1050] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db: attempting to promote NON_VOTER 50161459714942cb9b4ae7743f76d07d to VOTER
I20240501 14:16:07.008128 16061 consensus_queue.cc:237] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } }
I20240501 14:16:07.009238 14548 raft_consensus.cc:1261] T 84e2150d96c4451d95302894342228e8 P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Refusing update from remote peer c87cbbf2d3ab4143815ef2c74ae760db: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:16:07.009236 14418 raft_consensus.cc:1261] T 84e2150d96c4451d95302894342228e8 P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Refusing update from remote peer c87cbbf2d3ab4143815ef2c74ae760db: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:16:07.009567 15168 raft_consensus.cc:1261] T 84e2150d96c4451d95302894342228e8 P 50161459714942cb9b4ae7743f76d07d [term 1 LEARNER]: Refusing update from remote peer c87cbbf2d3ab4143815ef2c74ae760db: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:16:07.009595 16038 consensus_queue.cc:1035] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db [LEADER]: Connected to new peer: Peer: permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:16:07.009964 16038 consensus_queue.cc:1035] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db [LEADER]: Connected to new peer: Peer: permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:16:07.010123 15809 consensus_queue.cc:1035] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db [LEADER]: Connected to new peer: Peer: permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:16:07.011801 16037 raft_consensus.cc:2932] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 50161459714942cb9b4ae7743f76d07d (127.7.39.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } }
I20240501 14:16:07.012207 14548 raft_consensus.cc:2932] T 84e2150d96c4451d95302894342228e8 P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 50161459714942cb9b4ae7743f76d07d (127.7.39.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } }
I20240501 14:16:07.013657 14418 raft_consensus.cc:2932] T 84e2150d96c4451d95302894342228e8 P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 50161459714942cb9b4ae7743f76d07d (127.7.39.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } }
I20240501 14:16:07.013717 15168 raft_consensus.cc:2932] T 84e2150d96c4451d95302894342228e8 P 50161459714942cb9b4ae7743f76d07d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 50161459714942cb9b4ae7743f76d07d (127.7.39.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } }
I20240501 14:16:07.014417 14138 catalog_manager.cc:5484] T 84e2150d96c4451d95302894342228e8 P e625ab1b03d14e2aae194052d2c67bc6 reported cstate change: config changed from index 4 to 5, 50161459714942cb9b4ae7743f76d07d (127.7.39.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } }
I20240501 14:16:07.157505 14286 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "84e2150d96c4451d95302894342228e8"
dest_uuid: "c87cbbf2d3ab4143815ef2c74ae760db"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35396
I20240501 14:16:07.157685 14286 raft_consensus.cc:595] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 LEADER]: Received request to transfer leadership
I20240501 14:16:07.167325 14286 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3ded831efb0846a192400e8856dbd9b6"
dest_uuid: "c87cbbf2d3ab4143815ef2c74ae760db"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35424
I20240501 14:16:07.167445 14286 raft_consensus.cc:595] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 LEADER]: Received request to transfer leadership
I20240501 14:16:07.377821 14286 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "84e2150d96c4451d95302894342228e8"
dest_uuid: "c87cbbf2d3ab4143815ef2c74ae760db"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35430
I20240501 14:16:07.377965 14286 raft_consensus.cc:595] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 LEADER]: Received request to transfer leadership
I20240501 14:16:07.383023 16038 raft_consensus.cc:979] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db: : Instructing follower e625ab1b03d14e2aae194052d2c67bc6 to start an election
I20240501 14:16:07.383160 16038 raft_consensus.cc:1067] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 LEADER]: Signalling peer e625ab1b03d14e2aae194052d2c67bc6 to start an election
I20240501 14:16:07.383719 14548 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3ded831efb0846a192400e8856dbd9b6"
dest_uuid: "e625ab1b03d14e2aae194052d2c67bc6"
from {username='slave'} at 127.7.39.129:56501
I20240501 14:16:07.383883 14548 raft_consensus.cc:482] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:16:07.383949 14548 raft_consensus.cc:3037] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:16:07.384738 14548 raft_consensus.cc:504] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } }
I20240501 14:16:07.385113 14548 leader_election.cc:288] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6 [CANDIDATE]: Term 2 election: Requested vote from peers 17062e38120042aaaca05e153b2c6ccf (127.7.39.130:43603), c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129:42369), e3770e9581534f77a7506253b8d201b0 (127.7.39.132:38927)
I20240501 14:16:07.385440 14286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ded831efb0846a192400e8856dbd9b6" candidate_uuid: "e625ab1b03d14e2aae194052d2c67bc6" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "c87cbbf2d3ab4143815ef2c74ae760db"
I20240501 14:16:07.385447 14418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ded831efb0846a192400e8856dbd9b6" candidate_uuid: "e625ab1b03d14e2aae194052d2c67bc6" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "17062e38120042aaaca05e153b2c6ccf"
I20240501 14:16:07.385555 14286 raft_consensus.cc:3032] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:16:07.385587 14286 raft_consensus.cc:729] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c87cbbf2d3ab4143815ef2c74ae760db, State: Running, Role: LEADER
I20240501 14:16:07.385577 14418 raft_consensus.cc:3037] T 3ded831efb0846a192400e8856dbd9b6 P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:16:07.385586 15298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ded831efb0846a192400e8856dbd9b6" candidate_uuid: "e625ab1b03d14e2aae194052d2c67bc6" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "e3770e9581534f77a7506253b8d201b0"
I20240501 14:16:07.385757 15298 raft_consensus.cc:3037] T 3ded831efb0846a192400e8856dbd9b6 P e3770e9581534f77a7506253b8d201b0 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:16:07.385731 14286 consensus_queue.cc:260] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } }
I20240501 14:16:07.385910 14286 raft_consensus.cc:3037] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:16:07.386569 14418 raft_consensus.cc:2454] T 3ded831efb0846a192400e8856dbd9b6 P 17062e38120042aaaca05e153b2c6ccf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e625ab1b03d14e2aae194052d2c67bc6 in term 2.
I20240501 14:16:07.386639 14286 raft_consensus.cc:2454] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e625ab1b03d14e2aae194052d2c67bc6 in term 2.
I20240501 14:16:07.386811 15298 raft_consensus.cc:2454] T 3ded831efb0846a192400e8856dbd9b6 P e3770e9581534f77a7506253b8d201b0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e625ab1b03d14e2aae194052d2c67bc6 in term 2.
I20240501 14:16:07.387153 14483 leader_election.cc:302] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 17062e38120042aaaca05e153b2c6ccf, c87cbbf2d3ab4143815ef2c74ae760db, e625ab1b03d14e2aae194052d2c67bc6; no voters:
I20240501 14:16:07.387357 15734 raft_consensus.cc:2781] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:16:07.387514 15734 raft_consensus.cc:686] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6 [term 2 LEADER]: Becoming Leader. State: Replica: e625ab1b03d14e2aae194052d2c67bc6, State: Running, Role: LEADER
I20240501 14:16:07.387681 15734 consensus_queue.cc:237] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } }
I20240501 14:16:07.389590 14136 catalog_manager.cc:5484] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6 reported cstate change: term changed from 1 to 2, leader changed from c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129) to e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131). New cstate: current_term: 2 leader_uuid: "e625ab1b03d14e2aae194052d2c67bc6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:16:07.399582 15809 raft_consensus.cc:979] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db: : Instructing follower 17062e38120042aaaca05e153b2c6ccf to start an election
I20240501 14:16:07.399678 15809 raft_consensus.cc:1067] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 LEADER]: Signalling peer 17062e38120042aaaca05e153b2c6ccf to start an election
I20240501 14:16:07.400151 14418 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "84e2150d96c4451d95302894342228e8"
dest_uuid: "17062e38120042aaaca05e153b2c6ccf"
from {username='slave'} at 127.7.39.129:48011
I20240501 14:16:07.400285 14418 raft_consensus.cc:482] T 84e2150d96c4451d95302894342228e8 P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:16:07.400331 14418 raft_consensus.cc:3037] T 84e2150d96c4451d95302894342228e8 P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:16:07.401026 14418 raft_consensus.cc:504] T 84e2150d96c4451d95302894342228e8 P 17062e38120042aaaca05e153b2c6ccf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } }
I20240501 14:16:07.401437 14418 leader_election.cc:288] T 84e2150d96c4451d95302894342228e8 P 17062e38120042aaaca05e153b2c6ccf [CANDIDATE]: Term 2 election: Requested vote from peers c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129:42369), e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131:33199), 50161459714942cb9b4ae7743f76d07d (127.7.39.133:36729)
I20240501 14:16:07.401759 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84e2150d96c4451d95302894342228e8" candidate_uuid: "17062e38120042aaaca05e153b2c6ccf" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "e625ab1b03d14e2aae194052d2c67bc6"
I20240501 14:16:07.401805 14286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84e2150d96c4451d95302894342228e8" candidate_uuid: "17062e38120042aaaca05e153b2c6ccf" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "c87cbbf2d3ab4143815ef2c74ae760db"
I20240501 14:16:07.401878 14548 raft_consensus.cc:3037] T 84e2150d96c4451d95302894342228e8 P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:16:07.401904 14286 raft_consensus.cc:3032] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:16:07.401871 15168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84e2150d96c4451d95302894342228e8" candidate_uuid: "17062e38120042aaaca05e153b2c6ccf" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "50161459714942cb9b4ae7743f76d07d"
I20240501 14:16:07.401942 14286 raft_consensus.cc:729] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c87cbbf2d3ab4143815ef2c74ae760db, State: Running, Role: LEADER
I20240501 14:16:07.402082 15168 raft_consensus.cc:3037] T 84e2150d96c4451d95302894342228e8 P 50161459714942cb9b4ae7743f76d07d [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:16:07.402101 14286 consensus_queue.cc:260] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } }
I20240501 14:16:07.402277 14286 raft_consensus.cc:3037] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:16:07.402714 14548 raft_consensus.cc:2454] T 84e2150d96c4451d95302894342228e8 P e625ab1b03d14e2aae194052d2c67bc6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17062e38120042aaaca05e153b2c6ccf in term 2.
I20240501 14:16:07.403209 14286 raft_consensus.cc:2454] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17062e38120042aaaca05e153b2c6ccf in term 2.
I20240501 14:16:07.403218 15168 raft_consensus.cc:2454] T 84e2150d96c4451d95302894342228e8 P 50161459714942cb9b4ae7743f76d07d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17062e38120042aaaca05e153b2c6ccf in term 2.
I20240501 14:16:07.403519 14351 leader_election.cc:302] T 84e2150d96c4451d95302894342228e8 P 17062e38120042aaaca05e153b2c6ccf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 17062e38120042aaaca05e153b2c6ccf, c87cbbf2d3ab4143815ef2c74ae760db, e625ab1b03d14e2aae194052d2c67bc6; no voters:
I20240501 14:16:07.403708 15733 raft_consensus.cc:2781] T 84e2150d96c4451d95302894342228e8 P 17062e38120042aaaca05e153b2c6ccf [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:16:07.403882 15733 raft_consensus.cc:686] T 84e2150d96c4451d95302894342228e8 P 17062e38120042aaaca05e153b2c6ccf [term 2 LEADER]: Becoming Leader. State: Replica: 17062e38120042aaaca05e153b2c6ccf, State: Running, Role: LEADER
I20240501 14:16:07.404019 15733 consensus_queue.cc:237] T 84e2150d96c4451d95302894342228e8 P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } }
I20240501 14:16:07.405740 14136 catalog_manager.cc:5484] T 84e2150d96c4451d95302894342228e8 P 17062e38120042aaaca05e153b2c6ccf reported cstate change: term changed from 1 to 2, leader changed from c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129) to 17062e38120042aaaca05e153b2c6ccf (127.7.39.130). New cstate: current_term: 2 leader_uuid: "17062e38120042aaaca05e153b2c6ccf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:16:07.826038 14286 raft_consensus.cc:1261] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db [term 2 FOLLOWER]: Refusing update from remote peer e625ab1b03d14e2aae194052d2c67bc6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:16:07.826550 15734 consensus_queue.cc:1035] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:16:07.869603 15168 raft_consensus.cc:1261] T 84e2150d96c4451d95302894342228e8 P 50161459714942cb9b4ae7743f76d07d [term 2 FOLLOWER]: Refusing update from remote peer 17062e38120042aaaca05e153b2c6ccf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:16:07.870056 15733 consensus_queue.cc:1035] T 84e2150d96c4451d95302894342228e8 P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:16:07.872207 14286 raft_consensus.cc:1261] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db [term 2 FOLLOWER]: Refusing update from remote peer 17062e38120042aaaca05e153b2c6ccf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:16:07.872680 16109 consensus_queue.cc:1035] T 84e2150d96c4451d95302894342228e8 P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:16:07.875490 14548 raft_consensus.cc:1261] T 84e2150d96c4451d95302894342228e8 P e625ab1b03d14e2aae194052d2c67bc6 [term 2 FOLLOWER]: Refusing update from remote peer 17062e38120042aaaca05e153b2c6ccf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:16:07.876077 16135 consensus_queue.cc:1035] T 84e2150d96c4451d95302894342228e8 P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:16:07.896994 15298 raft_consensus.cc:1261] T 3ded831efb0846a192400e8856dbd9b6 P e3770e9581534f77a7506253b8d201b0 [term 2 FOLLOWER]: Refusing update from remote peer e625ab1b03d14e2aae194052d2c67bc6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:16:07.897511 15734 consensus_queue.cc:1035] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:16:07.900436 14418 raft_consensus.cc:1261] T 3ded831efb0846a192400e8856dbd9b6 P 17062e38120042aaaca05e153b2c6ccf [term 2 FOLLOWER]: Refusing update from remote peer e625ab1b03d14e2aae194052d2c67bc6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:16:07.900853 16140 consensus_queue.cc:1035] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:16:08.283715 14548 consensus_queue.cc:237] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } }
I20240501 14:16:08.284973 14418 raft_consensus.cc:1261] T 3ded831efb0846a192400e8856dbd9b6 P 17062e38120042aaaca05e153b2c6ccf [term 2 FOLLOWER]: Refusing update from remote peer e625ab1b03d14e2aae194052d2c67bc6: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:16:08.285024 15298 raft_consensus.cc:1261] T 3ded831efb0846a192400e8856dbd9b6 P e3770e9581534f77a7506253b8d201b0 [term 2 FOLLOWER]: Refusing update from remote peer e625ab1b03d14e2aae194052d2c67bc6: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:16:08.285454 16141 consensus_queue.cc:1035] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:16:08.285709 16140 consensus_queue.cc:1035] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:16:08.287454 16141 raft_consensus.cc:2932] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } }
I20240501 14:16:08.287863 15298 raft_consensus.cc:2932] T 3ded831efb0846a192400e8856dbd9b6 P e3770e9581534f77a7506253b8d201b0 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } }
I20240501 14:16:08.288030 14418 raft_consensus.cc:2932] T 3ded831efb0846a192400e8856dbd9b6 P 17062e38120042aaaca05e153b2c6ccf [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } }
I20240501 14:16:08.289064 14131 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3ded831efb0846a192400e8856dbd9b6 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:16:08.290638 14139 catalog_manager.cc:5484] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6 reported cstate change: config changed from index 5 to 7, VOTER c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129) evicted. New cstate: current_term: 2 leader_uuid: "e625ab1b03d14e2aae194052d2c67bc6" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:16:08.293388 14264 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ded831efb0846a192400e8856dbd9b6 with delete_type TABLET_DATA_TOMBSTONED (TS c87cbbf2d3ab4143815ef2c74ae760db not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:51888
I20240501 14:16:08.293885 16166 tablet_replica.cc:330] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db: stopping tablet replica
I20240501 14:16:08.294030 16166 raft_consensus.cc:2229] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:16:08.294135 16166 raft_consensus.cc:2258] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:16:08.294704 16166 ts_tablet_manager.cc:1875] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:16:08.296571 16166 ts_tablet_manager.cc:1888] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 14:16:08.296650 16166 log.cc:1196] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal/wals/3ded831efb0846a192400e8856dbd9b6
I20240501 14:16:08.297168 14128 catalog_manager.cc:4830] TS c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129:42369): tablet 3ded831efb0846a192400e8856dbd9b6 (table rebalance_test_table_0 [id=066c15421d0b476bbe4204501666d610]) successfully deleted
I20240501 14:16:08.472434 14418 consensus_queue.cc:237] T 84e2150d96c4451d95302894342228e8 P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } }
I20240501 14:16:08.473680 14548 raft_consensus.cc:1261] T 84e2150d96c4451d95302894342228e8 P e625ab1b03d14e2aae194052d2c67bc6 [term 2 FOLLOWER]: Refusing update from remote peer 17062e38120042aaaca05e153b2c6ccf: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:16:08.473709 15168 raft_consensus.cc:1261] T 84e2150d96c4451d95302894342228e8 P 50161459714942cb9b4ae7743f76d07d [term 2 FOLLOWER]: Refusing update from remote peer 17062e38120042aaaca05e153b2c6ccf: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:16:08.474117 16183 consensus_queue.cc:1035] T 84e2150d96c4451d95302894342228e8 P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:16:08.474323 16134 consensus_queue.cc:1035] T 84e2150d96c4451d95302894342228e8 P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:16:08.475944 16183 raft_consensus.cc:2932] T 84e2150d96c4451d95302894342228e8 P 17062e38120042aaaca05e153b2c6ccf [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } }
I20240501 14:16:08.476442 15168 raft_consensus.cc:2932] T 84e2150d96c4451d95302894342228e8 P 50161459714942cb9b4ae7743f76d07d [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } }
I20240501 14:16:08.476491 14548 raft_consensus.cc:2932] T 84e2150d96c4451d95302894342228e8 P e625ab1b03d14e2aae194052d2c67bc6 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } }
I20240501 14:16:08.477437 14129 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 84e2150d96c4451d95302894342228e8 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:16:08.478487 14136 catalog_manager.cc:5484] T 84e2150d96c4451d95302894342228e8 P 17062e38120042aaaca05e153b2c6ccf reported cstate change: config changed from index 5 to 7, VOTER c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129) evicted. New cstate: current_term: 2 leader_uuid: "17062e38120042aaaca05e153b2c6ccf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:16:08.481300 14264 tablet_service.cc:1514] Processing DeleteTablet for tablet 84e2150d96c4451d95302894342228e8 with delete_type TABLET_DATA_TOMBSTONED (TS c87cbbf2d3ab4143815ef2c74ae760db not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:51888
I20240501 14:16:08.481554 16166 tablet_replica.cc:330] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db: stopping tablet replica
I20240501 14:16:08.481717 16166 raft_consensus.cc:2229] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:16:08.481825 16166 raft_consensus.cc:2258] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:16:08.482543 16166 ts_tablet_manager.cc:1875] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:16:08.484272 16166 ts_tablet_manager.cc:1888] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 14:16:08.484342 16166 log.cc:1196] T 84e2150d96c4451d95302894342228e8 P c87cbbf2d3ab4143815ef2c74ae760db: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal/wals/84e2150d96c4451d95302894342228e8
I20240501 14:16:08.484823 14128 catalog_manager.cc:4830] TS c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129:42369): tablet 84e2150d96c4451d95302894342228e8 (table rebalance_test_table_0 [id=066c15421d0b476bbe4204501666d610]) successfully deleted
I20240501 14:16:08.714910 15148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:16:08.715642 14397 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:16:08.719319 15279 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:16:08.719324 14264 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:16:08.726704 14528 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:16:08.811983 15148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:16:08.812616 14264 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:16:08.815795 15279 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:16:08.818493 14397 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:16:08.827915 14528 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:16:08.857125 14418 consensus_queue.cc:237] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: NON_VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { promote: true } }
I20240501 14:16:08.858628 15298 raft_consensus.cc:1261] T 5f2b92ee1a1c46bcb36bc24563886d18 P e3770e9581534f77a7506253b8d201b0 [term 1 FOLLOWER]: Refusing update from remote peer 17062e38120042aaaca05e153b2c6ccf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:16:08.858724 14548 raft_consensus.cc:1261] T 5f2b92ee1a1c46bcb36bc24563886d18 P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Refusing update from remote peer 17062e38120042aaaca05e153b2c6ccf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 14:16:08.859081 14351 consensus_peers.cc:485] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf -> Peer c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129:42369): Couldn't send request to peer c87cbbf2d3ab4143815ef2c74ae760db. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:16:08.859215 16134 consensus_queue.cc:1035] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:16:08.859431 16183 consensus_queue.cc:1035] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:16:08.861115 16134 raft_consensus.cc:2932] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: NON_VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { promote: true } } }
I20240501 14:16:08.861609 14548 raft_consensus.cc:2932] T 5f2b92ee1a1c46bcb36bc24563886d18 P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: NON_VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { promote: true } } }
I20240501 14:16:08.861872 15298 raft_consensus.cc:2932] T 5f2b92ee1a1c46bcb36bc24563886d18 P e3770e9581534f77a7506253b8d201b0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: NON_VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { promote: true } } }
I20240501 14:16:08.864156 14141 catalog_manager.cc:5484] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf reported cstate change: config changed from index 6 to 7, NON_VOTER c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129) added. New cstate: current_term: 1 leader_uuid: "17062e38120042aaaca05e153b2c6ccf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: NON_VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:16:08.871001 14548 consensus_queue.cc:237] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: NON_VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { promote: true } }
I20240501 14:16:08.872375 15298 raft_consensus.cc:1261] T 3ded831efb0846a192400e8856dbd9b6 P e3770e9581534f77a7506253b8d201b0 [term 2 FOLLOWER]: Refusing update from remote peer e625ab1b03d14e2aae194052d2c67bc6: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
W20240501 14:16:08.872567 14483 consensus_peers.cc:485] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6 -> Peer c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129:42369): Couldn't send request to peer c87cbbf2d3ab4143815ef2c74ae760db. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:16:08.872738 14418 raft_consensus.cc:1261] T 3ded831efb0846a192400e8856dbd9b6 P 17062e38120042aaaca05e153b2c6ccf [term 2 FOLLOWER]: Refusing update from remote peer e625ab1b03d14e2aae194052d2c67bc6: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 14:16:08.872793 16276 consensus_queue.cc:1035] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:16:08.873149 16275 consensus_queue.cc:1035] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:16:08.874558 16276 raft_consensus.cc:2932] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: NON_VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { promote: true } } }
I20240501 14:16:08.874969 15298 raft_consensus.cc:2932] T 3ded831efb0846a192400e8856dbd9b6 P e3770e9581534f77a7506253b8d201b0 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: NON_VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { promote: true } } }
I20240501 14:16:08.875191 14418 raft_consensus.cc:2932] T 3ded831efb0846a192400e8856dbd9b6 P 17062e38120042aaaca05e153b2c6ccf [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: NON_VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { promote: true } } }
I20240501 14:16:08.876914 14139 catalog_manager.cc:5484] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6 reported cstate change: config changed from index 7 to 8, NON_VOTER c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129) added. New cstate: current_term: 2 leader_uuid: "e625ab1b03d14e2aae194052d2c67bc6" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: NON_VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:16:08.885202 14418 consensus_queue.cc:237] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: NON_VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { promote: true } }
I20240501 14:16:08.886484 14286 raft_consensus.cc:1261] T 40732c47173c429aa69c6a41475849de P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Refusing update from remote peer 17062e38120042aaaca05e153b2c6ccf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:16:08.886586 15168 raft_consensus.cc:1261] T 40732c47173c429aa69c6a41475849de P 50161459714942cb9b4ae7743f76d07d [term 1 FOLLOWER]: Refusing update from remote peer 17062e38120042aaaca05e153b2c6ccf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 14:16:08.886610 14354 consensus_peers.cc:485] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf -> Peer e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131:33199): Couldn't send request to peer e625ab1b03d14e2aae194052d2c67bc6. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:16:08.886898 16183 consensus_queue.cc:1035] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:16:08.887117 16134 consensus_queue.cc:1035] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:16:08.889024 16182 raft_consensus.cc:2932] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: NON_VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { promote: true } } }
I20240501 14:16:08.889453 15168 raft_consensus.cc:2932] T 40732c47173c429aa69c6a41475849de P 50161459714942cb9b4ae7743f76d07d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: NON_VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { promote: true } } }
I20240501 14:16:08.891316 14286 raft_consensus.cc:2932] T 40732c47173c429aa69c6a41475849de P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: NON_VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { promote: true } } }
I20240501 14:16:08.891356 14136 catalog_manager.cc:5484] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf reported cstate change: config changed from index 6 to 7, NON_VOTER e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131) added. New cstate: current_term: 1 leader_uuid: "17062e38120042aaaca05e153b2c6ccf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: NON_VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:16:08.899556 14548 consensus_queue.cc:237] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: NON_VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { promote: true } }
I20240501 14:16:08.900966 15168 raft_consensus.cc:1261] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 50161459714942cb9b4ae7743f76d07d [term 1 FOLLOWER]: Refusing update from remote peer e625ab1b03d14e2aae194052d2c67bc6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:16:08.901021 14286 raft_consensus.cc:1261] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Refusing update from remote peer e625ab1b03d14e2aae194052d2c67bc6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 14:16:08.901237 14484 consensus_peers.cc:485] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6 -> Peer 17062e38120042aaaca05e153b2c6ccf (127.7.39.130:43603): Couldn't send request to peer 17062e38120042aaaca05e153b2c6ccf. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:16:08.901361 15734 consensus_queue.cc:1035] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:16:08.901542 16276 consensus_queue.cc:1035] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:16:08.903259 16275 raft_consensus.cc:2932] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 17062e38120042aaaca05e153b2c6ccf (127.7.39.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: NON_VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { promote: true } } }
I20240501 14:16:08.903667 15168 raft_consensus.cc:2932] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 50161459714942cb9b4ae7743f76d07d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 17062e38120042aaaca05e153b2c6ccf (127.7.39.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: NON_VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { promote: true } } }
I20240501 14:16:08.904911 14286 raft_consensus.cc:2932] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 17062e38120042aaaca05e153b2c6ccf (127.7.39.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: NON_VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { promote: true } } }
I20240501 14:16:08.905941 14141 catalog_manager.cc:5484] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 50161459714942cb9b4ae7743f76d07d reported cstate change: config changed from index 6 to 7, NON_VOTER 17062e38120042aaaca05e153b2c6ccf (127.7.39.130) added. New cstate: current_term: 1 leader_uuid: "e625ab1b03d14e2aae194052d2c67bc6" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: NON_VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { promote: true } } }
I20240501 14:16:09.305326 16333 ts_tablet_manager.cc:902] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db: Initiating tablet copy from peer e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131:33199)
I20240501 14:16:09.306041 16333 tablet_copy_client.cc:308] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.131:33199
I20240501 14:16:09.306679 14570 tablet_copy_service.cc:140] P e625ab1b03d14e2aae194052d2c67bc6: Received BeginTabletCopySession request for tablet 3ded831efb0846a192400e8856dbd9b6 from peer c87cbbf2d3ab4143815ef2c74ae760db ({username='slave'} at 127.7.39.129:56501)
I20240501 14:16:09.306780 14570 tablet_copy_service.cc:161] P e625ab1b03d14e2aae194052d2c67bc6: Beginning new tablet copy session on tablet 3ded831efb0846a192400e8856dbd9b6 from peer c87cbbf2d3ab4143815ef2c74ae760db at {username='slave'} at 127.7.39.129:56501: session id = c87cbbf2d3ab4143815ef2c74ae760db-3ded831efb0846a192400e8856dbd9b6
I20240501 14:16:09.307600 14570 tablet_copy_source_session.cc:205] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:16:09.308529 16333 ts_tablet_manager.cc:1875] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:16:09.310523 16333 ts_tablet_manager.cc:1888] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.6
I20240501 14:16:09.310699 16333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ded831efb0846a192400e8856dbd9b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:09.311717 16333 tablet_copy_client.cc:783] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db: tablet copy: Starting download of 0 data blocks...
I20240501 14:16:09.311929 16333 tablet_copy_client.cc:647] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db: tablet copy: Starting download of 1 WAL segments...
I20240501 14:16:09.313326 16333 tablet_copy_client.cc:523] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:16:09.314574 16333 tablet_bootstrap.cc:492] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db: Bootstrap starting.
I20240501 14:16:09.324096 16339 ts_tablet_manager.cc:902] T 40732c47173c429aa69c6a41475849de P e625ab1b03d14e2aae194052d2c67bc6: Initiating tablet copy from peer 17062e38120042aaaca05e153b2c6ccf (127.7.39.130:43603)
I20240501 14:16:09.324883 16339 tablet_copy_client.cc:308] T 40732c47173c429aa69c6a41475849de P e625ab1b03d14e2aae194052d2c67bc6: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.130:43603
I20240501 14:16:09.325441 14438 tablet_copy_service.cc:140] P 17062e38120042aaaca05e153b2c6ccf: Received BeginTabletCopySession request for tablet 40732c47173c429aa69c6a41475849de from peer e625ab1b03d14e2aae194052d2c67bc6 ({username='slave'} at 127.7.39.131:59181)
I20240501 14:16:09.325554 14438 tablet_copy_service.cc:161] P 17062e38120042aaaca05e153b2c6ccf: Beginning new tablet copy session on tablet 40732c47173c429aa69c6a41475849de from peer e625ab1b03d14e2aae194052d2c67bc6 at {username='slave'} at 127.7.39.131:59181: session id = e625ab1b03d14e2aae194052d2c67bc6-40732c47173c429aa69c6a41475849de
I20240501 14:16:09.326256 14438 tablet_copy_source_session.cc:205] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:16:09.327167 16339 ts_tablet_manager.cc:1875] T 40732c47173c429aa69c6a41475849de P e625ab1b03d14e2aae194052d2c67bc6: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:16:09.329228 16339 ts_tablet_manager.cc:1888] T 40732c47173c429aa69c6a41475849de P e625ab1b03d14e2aae194052d2c67bc6: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 14:16:09.329399 16339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40732c47173c429aa69c6a41475849de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:09.330528 16339 tablet_copy_client.cc:783] T 40732c47173c429aa69c6a41475849de P e625ab1b03d14e2aae194052d2c67bc6: tablet copy: Starting download of 0 data blocks...
I20240501 14:16:09.330744 16339 tablet_copy_client.cc:647] T 40732c47173c429aa69c6a41475849de P e625ab1b03d14e2aae194052d2c67bc6: tablet copy: Starting download of 1 WAL segments...
I20240501 14:16:09.331985 16339 tablet_copy_client.cc:523] T 40732c47173c429aa69c6a41475849de P e625ab1b03d14e2aae194052d2c67bc6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:16:09.333454 16339 tablet_bootstrap.cc:492] T 40732c47173c429aa69c6a41475849de P e625ab1b03d14e2aae194052d2c67bc6: Bootstrap starting.
I20240501 14:16:09.336562 16333 tablet_bootstrap.cc:492] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:16:09.336839 16333 tablet_bootstrap.cc:492] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db: Bootstrap complete.
I20240501 14:16:09.337039 16333 ts_tablet_manager.cc:1367] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db: Time spent bootstrapping tablet: real 0.022s user 0.022s sys 0.000s
I20240501 14:16:09.337460 16333 raft_consensus.cc:348] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: NON_VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { promote: true } }
I20240501 14:16:09.337589 16333 raft_consensus.cc:729] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: c87cbbf2d3ab4143815ef2c74ae760db, State: Initialized, Role: LEARNER
I20240501 14:16:09.337698 16333 consensus_queue.cc:260] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: NON_VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { promote: true } }
I20240501 14:16:09.338023 16333 ts_tablet_manager.cc:1398] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:16:09.338629 14570 tablet_copy_service.cc:342] P e625ab1b03d14e2aae194052d2c67bc6: Request end of tablet copy session c87cbbf2d3ab4143815ef2c74ae760db-3ded831efb0846a192400e8856dbd9b6 received from {username='slave'} at 127.7.39.129:56501
I20240501 14:16:09.338706 14570 tablet_copy_service.cc:433] P e625ab1b03d14e2aae194052d2c67bc6: ending tablet copy session c87cbbf2d3ab4143815ef2c74ae760db-3ded831efb0846a192400e8856dbd9b6 on tablet 3ded831efb0846a192400e8856dbd9b6 with peer c87cbbf2d3ab4143815ef2c74ae760db
I20240501 14:16:09.352353 16339 tablet_bootstrap.cc:492] T 40732c47173c429aa69c6a41475849de P e625ab1b03d14e2aae194052d2c67bc6: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:16:09.352620 16339 tablet_bootstrap.cc:492] T 40732c47173c429aa69c6a41475849de P e625ab1b03d14e2aae194052d2c67bc6: Bootstrap complete.
I20240501 14:16:09.352764 16339 ts_tablet_manager.cc:1367] T 40732c47173c429aa69c6a41475849de P e625ab1b03d14e2aae194052d2c67bc6: Time spent bootstrapping tablet: real 0.019s user 0.019s sys 0.000s
I20240501 14:16:09.353197 16339 raft_consensus.cc:348] T 40732c47173c429aa69c6a41475849de P e625ab1b03d14e2aae194052d2c67bc6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: NON_VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { promote: true } }
I20240501 14:16:09.353293 16339 raft_consensus.cc:729] T 40732c47173c429aa69c6a41475849de P e625ab1b03d14e2aae194052d2c67bc6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e625ab1b03d14e2aae194052d2c67bc6, State: Initialized, Role: LEARNER
I20240501 14:16:09.353374 16339 consensus_queue.cc:260] T 40732c47173c429aa69c6a41475849de P e625ab1b03d14e2aae194052d2c67bc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: NON_VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { promote: true } }
I20240501 14:16:09.353631 16339 ts_tablet_manager.cc:1398] T 40732c47173c429aa69c6a41475849de P e625ab1b03d14e2aae194052d2c67bc6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:16:09.354084 14438 tablet_copy_service.cc:342] P 17062e38120042aaaca05e153b2c6ccf: Request end of tablet copy session e625ab1b03d14e2aae194052d2c67bc6-40732c47173c429aa69c6a41475849de received from {username='slave'} at 127.7.39.131:59181
I20240501 14:16:09.354163 14438 tablet_copy_service.cc:433] P 17062e38120042aaaca05e153b2c6ccf: ending tablet copy session e625ab1b03d14e2aae194052d2c67bc6-40732c47173c429aa69c6a41475849de on tablet 40732c47173c429aa69c6a41475849de with peer e625ab1b03d14e2aae194052d2c67bc6
I20240501 14:16:09.377341 16333 ts_tablet_manager.cc:902] T 5f2b92ee1a1c46bcb36bc24563886d18 P c87cbbf2d3ab4143815ef2c74ae760db: Initiating tablet copy from peer 17062e38120042aaaca05e153b2c6ccf (127.7.39.130:43603)
I20240501 14:16:09.377848 16333 tablet_copy_client.cc:308] T 5f2b92ee1a1c46bcb36bc24563886d18 P c87cbbf2d3ab4143815ef2c74ae760db: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.130:43603
I20240501 14:16:09.378374 14438 tablet_copy_service.cc:140] P 17062e38120042aaaca05e153b2c6ccf: Received BeginTabletCopySession request for tablet 5f2b92ee1a1c46bcb36bc24563886d18 from peer c87cbbf2d3ab4143815ef2c74ae760db ({username='slave'} at 127.7.39.129:48011)
I20240501 14:16:09.378458 14438 tablet_copy_service.cc:161] P 17062e38120042aaaca05e153b2c6ccf: Beginning new tablet copy session on tablet 5f2b92ee1a1c46bcb36bc24563886d18 from peer c87cbbf2d3ab4143815ef2c74ae760db at {username='slave'} at 127.7.39.129:48011: session id = c87cbbf2d3ab4143815ef2c74ae760db-5f2b92ee1a1c46bcb36bc24563886d18
I20240501 14:16:09.379132 14438 tablet_copy_source_session.cc:205] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:16:09.379743 16333 ts_tablet_manager.cc:1875] T 5f2b92ee1a1c46bcb36bc24563886d18 P c87cbbf2d3ab4143815ef2c74ae760db: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:16:09.382083 16333 ts_tablet_manager.cc:1888] T 5f2b92ee1a1c46bcb36bc24563886d18 P c87cbbf2d3ab4143815ef2c74ae760db: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 14:16:09.382257 16333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f2b92ee1a1c46bcb36bc24563886d18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:09.383143 16333 tablet_copy_client.cc:783] T 5f2b92ee1a1c46bcb36bc24563886d18 P c87cbbf2d3ab4143815ef2c74ae760db: tablet copy: Starting download of 0 data blocks...
I20240501 14:16:09.383280 16333 tablet_copy_client.cc:647] T 5f2b92ee1a1c46bcb36bc24563886d18 P c87cbbf2d3ab4143815ef2c74ae760db: tablet copy: Starting download of 1 WAL segments...
I20240501 14:16:09.384277 16333 tablet_copy_client.cc:523] T 5f2b92ee1a1c46bcb36bc24563886d18 P c87cbbf2d3ab4143815ef2c74ae760db: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:16:09.385187 16333 tablet_bootstrap.cc:492] T 5f2b92ee1a1c46bcb36bc24563886d18 P c87cbbf2d3ab4143815ef2c74ae760db: Bootstrap starting.
I20240501 14:16:09.402226 16333 tablet_bootstrap.cc:492] T 5f2b92ee1a1c46bcb36bc24563886d18 P c87cbbf2d3ab4143815ef2c74ae760db: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:16:09.402503 16333 tablet_bootstrap.cc:492] T 5f2b92ee1a1c46bcb36bc24563886d18 P c87cbbf2d3ab4143815ef2c74ae760db: Bootstrap complete.
I20240501 14:16:09.402642 16333 ts_tablet_manager.cc:1367] T 5f2b92ee1a1c46bcb36bc24563886d18 P c87cbbf2d3ab4143815ef2c74ae760db: Time spent bootstrapping tablet: real 0.017s user 0.015s sys 0.000s
I20240501 14:16:09.402959 16333 raft_consensus.cc:348] T 5f2b92ee1a1c46bcb36bc24563886d18 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: NON_VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { promote: true } }
I20240501 14:16:09.403075 16333 raft_consensus.cc:729] T 5f2b92ee1a1c46bcb36bc24563886d18 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c87cbbf2d3ab4143815ef2c74ae760db, State: Initialized, Role: LEARNER
I20240501 14:16:09.403188 16333 consensus_queue.cc:260] T 5f2b92ee1a1c46bcb36bc24563886d18 P c87cbbf2d3ab4143815ef2c74ae760db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: NON_VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { promote: true } }
I20240501 14:16:09.403429 16333 ts_tablet_manager.cc:1398] T 5f2b92ee1a1c46bcb36bc24563886d18 P c87cbbf2d3ab4143815ef2c74ae760db: Time spent starting tablet: real 0.001s user 0.000s sys 0.004s
I20240501 14:16:09.403861 14438 tablet_copy_service.cc:342] P 17062e38120042aaaca05e153b2c6ccf: Request end of tablet copy session c87cbbf2d3ab4143815ef2c74ae760db-5f2b92ee1a1c46bcb36bc24563886d18 received from {username='slave'} at 127.7.39.129:48011
I20240501 14:16:09.403928 14438 tablet_copy_service.cc:433] P 17062e38120042aaaca05e153b2c6ccf: ending tablet copy session c87cbbf2d3ab4143815ef2c74ae760db-5f2b92ee1a1c46bcb36bc24563886d18 on tablet 5f2b92ee1a1c46bcb36bc24563886d18 with peer c87cbbf2d3ab4143815ef2c74ae760db
I20240501 14:16:09.464111 16370 ts_tablet_manager.cc:902] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 17062e38120042aaaca05e153b2c6ccf: Initiating tablet copy from peer e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131:33199)
I20240501 14:16:09.464874 16370 tablet_copy_client.cc:308] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 17062e38120042aaaca05e153b2c6ccf: tablet copy: Beginning tablet copy session from remote peer at address 127.7.39.131:33199
I20240501 14:16:09.465435 14570 tablet_copy_service.cc:140] P e625ab1b03d14e2aae194052d2c67bc6: Received BeginTabletCopySession request for tablet 47ef7ee4ea3b46cbbac2126ceaa5fa24 from peer 17062e38120042aaaca05e153b2c6ccf ({username='slave'} at 127.7.39.130:34013)
I20240501 14:16:09.465534 14570 tablet_copy_service.cc:161] P e625ab1b03d14e2aae194052d2c67bc6: Beginning new tablet copy session on tablet 47ef7ee4ea3b46cbbac2126ceaa5fa24 from peer 17062e38120042aaaca05e153b2c6ccf at {username='slave'} at 127.7.39.130:34013: session id = 17062e38120042aaaca05e153b2c6ccf-47ef7ee4ea3b46cbbac2126ceaa5fa24
I20240501 14:16:09.466241 14570 tablet_copy_source_session.cc:205] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:16:09.467073 16370 ts_tablet_manager.cc:1875] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 17062e38120042aaaca05e153b2c6ccf: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:16:09.468873 16370 ts_tablet_manager.cc:1888] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 17062e38120042aaaca05e153b2c6ccf: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 14:16:09.469017 16370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47ef7ee4ea3b46cbbac2126ceaa5fa24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:09.469897 16370 tablet_copy_client.cc:783] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 17062e38120042aaaca05e153b2c6ccf: tablet copy: Starting download of 0 data blocks...
I20240501 14:16:09.470064 16370 tablet_copy_client.cc:647] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 17062e38120042aaaca05e153b2c6ccf: tablet copy: Starting download of 1 WAL segments...
I20240501 14:16:09.471292 16370 tablet_copy_client.cc:523] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 17062e38120042aaaca05e153b2c6ccf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:16:09.472388 16370 tablet_bootstrap.cc:492] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 17062e38120042aaaca05e153b2c6ccf: Bootstrap starting.
I20240501 14:16:09.491550 16370 tablet_bootstrap.cc:492] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 17062e38120042aaaca05e153b2c6ccf: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:16:09.491770 16370 tablet_bootstrap.cc:492] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 17062e38120042aaaca05e153b2c6ccf: Bootstrap complete.
I20240501 14:16:09.491891 16370 ts_tablet_manager.cc:1367] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 17062e38120042aaaca05e153b2c6ccf: Time spent bootstrapping tablet: real 0.020s user 0.016s sys 0.004s
I20240501 14:16:09.492238 16370 raft_consensus.cc:348] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 17062e38120042aaaca05e153b2c6ccf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: NON_VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { promote: true } }
I20240501 14:16:09.492336 16370 raft_consensus.cc:729] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 17062e38120042aaaca05e153b2c6ccf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 17062e38120042aaaca05e153b2c6ccf, State: Initialized, Role: LEARNER
I20240501 14:16:09.492421 16370 consensus_queue.cc:260] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 17062e38120042aaaca05e153b2c6ccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: NON_VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { promote: true } }
I20240501 14:16:09.492657 16370 ts_tablet_manager.cc:1398] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 17062e38120042aaaca05e153b2c6ccf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:16:09.493135 14570 tablet_copy_service.cc:342] P e625ab1b03d14e2aae194052d2c67bc6: Request end of tablet copy session 17062e38120042aaaca05e153b2c6ccf-47ef7ee4ea3b46cbbac2126ceaa5fa24 received from {username='slave'} at 127.7.39.130:34013
I20240501 14:16:09.493227 14570 tablet_copy_service.cc:433] P e625ab1b03d14e2aae194052d2c67bc6: ending tablet copy session 17062e38120042aaaca05e153b2c6ccf-47ef7ee4ea3b46cbbac2126ceaa5fa24 on tablet 47ef7ee4ea3b46cbbac2126ceaa5fa24 with peer 17062e38120042aaaca05e153b2c6ccf
I20240501 14:16:09.743592 14286 raft_consensus.cc:1203] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240501 14:16:09.797160 14286 raft_consensus.cc:1203] T 5f2b92ee1a1c46bcb36bc24563886d18 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:16:09.803653 14548 raft_consensus.cc:1203] T 40732c47173c429aa69c6a41475849de P e625ab1b03d14e2aae194052d2c67bc6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:16:09.872175 14418 raft_consensus.cc:1203] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 17062e38120042aaaca05e153b2c6ccf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:16:10.256175 15734 raft_consensus.cc:1050] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6: attempting to promote NON_VOTER c87cbbf2d3ab4143815ef2c74ae760db to VOTER
I20240501 14:16:10.256671 15734 consensus_queue.cc:237] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { promote: false } }
I20240501 14:16:10.258261 14286 raft_consensus.cc:1261] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db [term 2 LEARNER]: Refusing update from remote peer e625ab1b03d14e2aae194052d2c67bc6: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:16:10.258261 14418 raft_consensus.cc:1261] T 3ded831efb0846a192400e8856dbd9b6 P 17062e38120042aaaca05e153b2c6ccf [term 2 FOLLOWER]: Refusing update from remote peer e625ab1b03d14e2aae194052d2c67bc6: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:16:10.258361 15298 raft_consensus.cc:1261] T 3ded831efb0846a192400e8856dbd9b6 P e3770e9581534f77a7506253b8d201b0 [term 2 FOLLOWER]: Refusing update from remote peer e625ab1b03d14e2aae194052d2c67bc6: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:16:10.258991 16451 consensus_queue.cc:1035] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:16:10.259235 15734 consensus_queue.cc:1035] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:16:10.259498 16396 consensus_queue.cc:1035] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:16:10.261956 16451 raft_consensus.cc:2932] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { promote: false } } }
I20240501 14:16:10.262425 15298 raft_consensus.cc:2932] T 3ded831efb0846a192400e8856dbd9b6 P e3770e9581534f77a7506253b8d201b0 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { promote: false } } }
I20240501 14:16:10.265602 14418 raft_consensus.cc:2932] T 3ded831efb0846a192400e8856dbd9b6 P 17062e38120042aaaca05e153b2c6ccf [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { promote: false } } }
I20240501 14:16:10.265610 14286 raft_consensus.cc:2932] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { promote: false } } }
I20240501 14:16:10.265851 14136 catalog_manager.cc:5484] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6 reported cstate change: config changed from index 8 to 9, c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "e625ab1b03d14e2aae194052d2c67bc6" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:16:10.266745 16182 raft_consensus.cc:1050] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf: attempting to promote NON_VOTER e625ab1b03d14e2aae194052d2c67bc6 to VOTER
I20240501 14:16:10.267210 16182 consensus_queue.cc:237] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { promote: false } }
I20240501 14:16:10.269604 14286 raft_consensus.cc:1261] T 40732c47173c429aa69c6a41475849de P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Refusing update from remote peer 17062e38120042aaaca05e153b2c6ccf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:16:10.269706 14548 raft_consensus.cc:1261] T 40732c47173c429aa69c6a41475849de P e625ab1b03d14e2aae194052d2c67bc6 [term 1 LEARNER]: Refusing update from remote peer 17062e38120042aaaca05e153b2c6ccf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:16:10.270077 16397 consensus_queue.cc:1035] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:16:10.271148 15168 raft_consensus.cc:1261] T 40732c47173c429aa69c6a41475849de P 50161459714942cb9b4ae7743f76d07d [term 1 FOLLOWER]: Refusing update from remote peer 17062e38120042aaaca05e153b2c6ccf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:16:10.271170 16403 consensus_queue.cc:1035] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:16:10.271667 16403 consensus_queue.cc:1035] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:16:10.273718 14548 consensus_queue.cc:237] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { promote: false } }
I20240501 14:16:10.274924 15298 raft_consensus.cc:1261] T 3ded831efb0846a192400e8856dbd9b6 P e3770e9581534f77a7506253b8d201b0 [term 2 FOLLOWER]: Refusing update from remote peer e625ab1b03d14e2aae194052d2c67bc6: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 14:16:10.275130 14286 raft_consensus.cc:1261] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db [term 2 FOLLOWER]: Refusing update from remote peer e625ab1b03d14e2aae194052d2c67bc6: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 14:16:10.275460 16461 consensus_queue.cc:1035] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:16:10.275759 16461 consensus_queue.cc:1035] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:16:10.276435 16468 raft_consensus.cc:2932] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { promote: false } } }
I20240501 14:16:10.277254 16450 raft_consensus.cc:2932] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 17062e38120042aaaca05e153b2c6ccf (127.7.39.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { promote: false } } }
I20240501 14:16:10.278357 14285 raft_consensus.cc:2932] T 40732c47173c429aa69c6a41475849de P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { promote: false } } }
I20240501 14:16:10.278898 14286 raft_consensus.cc:2932] T 3ded831efb0846a192400e8856dbd9b6 P c87cbbf2d3ab4143815ef2c74ae760db [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 17062e38120042aaaca05e153b2c6ccf (127.7.39.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { promote: false } } }
I20240501 14:16:10.279021 14131 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3ded831efb0846a192400e8856dbd9b6 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:16:10.279034 14139 catalog_manager.cc:5484] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf reported cstate change: config changed from index 7 to 8, e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "17062e38120042aaaca05e153b2c6ccf" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:16:10.280274 15298 raft_consensus.cc:2932] T 3ded831efb0846a192400e8856dbd9b6 P e3770e9581534f77a7506253b8d201b0 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 17062e38120042aaaca05e153b2c6ccf (127.7.39.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { promote: false } } }
I20240501 14:16:10.281350 14138 catalog_manager.cc:5484] T 3ded831efb0846a192400e8856dbd9b6 P e625ab1b03d14e2aae194052d2c67bc6 reported cstate change: config changed from index 9 to 10, VOTER 17062e38120042aaaca05e153b2c6ccf (127.7.39.130) evicted. New cstate: current_term: 2 leader_uuid: "e625ab1b03d14e2aae194052d2c67bc6" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:16:10.281746 14548 raft_consensus.cc:2932] T 40732c47173c429aa69c6a41475849de P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { promote: false } } }
I20240501 14:16:10.283398 15168 raft_consensus.cc:2932] T 40732c47173c429aa69c6a41475849de P 50161459714942cb9b4ae7743f76d07d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { promote: false } } }
I20240501 14:16:10.284660 14397 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ded831efb0846a192400e8856dbd9b6 with delete_type TABLET_DATA_TOMBSTONED (TS 17062e38120042aaaca05e153b2c6ccf not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:36870
I20240501 14:16:10.284988 16478 tablet_replica.cc:330] T 3ded831efb0846a192400e8856dbd9b6 P 17062e38120042aaaca05e153b2c6ccf: stopping tablet replica
I20240501 14:16:10.285501 16478 raft_consensus.cc:2229] T 3ded831efb0846a192400e8856dbd9b6 P 17062e38120042aaaca05e153b2c6ccf [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:16:10.285705 16478 raft_consensus.cc:2258] T 3ded831efb0846a192400e8856dbd9b6 P 17062e38120042aaaca05e153b2c6ccf [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:16:10.286265 14418 consensus_queue.cc:237] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { promote: false } }
I20240501 14:16:10.287432 15168 raft_consensus.cc:1261] T 40732c47173c429aa69c6a41475849de P 50161459714942cb9b4ae7743f76d07d [term 1 FOLLOWER]: Refusing update from remote peer 17062e38120042aaaca05e153b2c6ccf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:16:10.287431 14548 raft_consensus.cc:1261] T 40732c47173c429aa69c6a41475849de P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Refusing update from remote peer 17062e38120042aaaca05e153b2c6ccf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:16:10.287565 16478 ts_tablet_manager.cc:1875] T 3ded831efb0846a192400e8856dbd9b6 P 17062e38120042aaaca05e153b2c6ccf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:16:10.287797 16468 consensus_queue.cc:1035] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:16:10.287968 16403 consensus_queue.cc:1035] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:16:10.289491 16468 raft_consensus.cc:2932] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { promote: false } } }
I20240501 14:16:10.289866 16478 ts_tablet_manager.cc:1888] T 3ded831efb0846a192400e8856dbd9b6 P 17062e38120042aaaca05e153b2c6ccf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240501 14:16:10.289952 16478 log.cc:1196] T 3ded831efb0846a192400e8856dbd9b6 P 17062e38120042aaaca05e153b2c6ccf: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal/wals/3ded831efb0846a192400e8856dbd9b6
I20240501 14:16:10.290172 14548 raft_consensus.cc:2932] T 40732c47173c429aa69c6a41475849de P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { promote: false } } }
I20240501 14:16:10.290246 15168 raft_consensus.cc:2932] T 40732c47173c429aa69c6a41475849de P 50161459714942cb9b4ae7743f76d07d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { promote: false } } }
I20240501 14:16:10.290540 14129 catalog_manager.cc:4830] TS 17062e38120042aaaca05e153b2c6ccf (127.7.39.130:43603): tablet 3ded831efb0846a192400e8856dbd9b6 (table rebalance_test_table_0 [id=066c15421d0b476bbe4204501666d610]) successfully deleted
I20240501 14:16:10.291848 14129 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 40732c47173c429aa69c6a41475849de with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:16:10.292373 14141 catalog_manager.cc:5484] T 40732c47173c429aa69c6a41475849de P 17062e38120042aaaca05e153b2c6ccf reported cstate change: config changed from index 8 to 9, VOTER c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129) evicted. New cstate: current_term: 1 leader_uuid: "17062e38120042aaaca05e153b2c6ccf" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:16:10.294843 14264 tablet_service.cc:1514] Processing DeleteTablet for tablet 40732c47173c429aa69c6a41475849de with delete_type TABLET_DATA_TOMBSTONED (TS c87cbbf2d3ab4143815ef2c74ae760db not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:51888
I20240501 14:16:10.295208 16479 tablet_replica.cc:330] T 40732c47173c429aa69c6a41475849de P c87cbbf2d3ab4143815ef2c74ae760db: stopping tablet replica
I20240501 14:16:10.295327 16479 raft_consensus.cc:2229] T 40732c47173c429aa69c6a41475849de P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:16:10.295542 16479 raft_consensus.cc:2258] T 40732c47173c429aa69c6a41475849de P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:16:10.296116 16479 ts_tablet_manager.cc:1875] T 40732c47173c429aa69c6a41475849de P c87cbbf2d3ab4143815ef2c74ae760db: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:16:10.297719 16479 ts_tablet_manager.cc:1888] T 40732c47173c429aa69c6a41475849de P c87cbbf2d3ab4143815ef2c74ae760db: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 14:16:10.297781 16479 log.cc:1196] T 40732c47173c429aa69c6a41475849de P c87cbbf2d3ab4143815ef2c74ae760db: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal/wals/40732c47173c429aa69c6a41475849de
I20240501 14:16:10.298245 14128 catalog_manager.cc:4830] TS c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129:42369): tablet 40732c47173c429aa69c6a41475849de (table rebalance_test_table_1 [id=2938421fb092484186ed564e78a8fd87]) successfully deleted
I20240501 14:16:10.339856 16182 raft_consensus.cc:1050] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf: attempting to promote NON_VOTER c87cbbf2d3ab4143815ef2c74ae760db to VOTER
I20240501 14:16:10.340252 16182 consensus_queue.cc:237] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { promote: false } }
I20240501 14:16:10.341555 14548 raft_consensus.cc:1261] T 5f2b92ee1a1c46bcb36bc24563886d18 P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Refusing update from remote peer 17062e38120042aaaca05e153b2c6ccf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:16:10.341591 14286 raft_consensus.cc:1261] T 5f2b92ee1a1c46bcb36bc24563886d18 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 LEARNER]: Refusing update from remote peer 17062e38120042aaaca05e153b2c6ccf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:16:10.341599 15298 raft_consensus.cc:1261] T 5f2b92ee1a1c46bcb36bc24563886d18 P e3770e9581534f77a7506253b8d201b0 [term 1 FOLLOWER]: Refusing update from remote peer 17062e38120042aaaca05e153b2c6ccf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:16:10.341946 16468 consensus_queue.cc:1035] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:16:10.342132 16403 consensus_queue.cc:1035] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:16:10.342401 16182 consensus_queue.cc:1035] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:16:10.344702 16468 raft_consensus.cc:2932] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { promote: false } } }
I20240501 14:16:10.345149 14285 raft_consensus.cc:2932] T 5f2b92ee1a1c46bcb36bc24563886d18 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { promote: false } } }
I20240501 14:16:10.345217 15298 raft_consensus.cc:2932] T 5f2b92ee1a1c46bcb36bc24563886d18 P e3770e9581534f77a7506253b8d201b0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { promote: false } } }
I20240501 14:16:10.347338 14548 raft_consensus.cc:2932] T 5f2b92ee1a1c46bcb36bc24563886d18 P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { promote: false } } }
I20240501 14:16:10.347911 14138 catalog_manager.cc:5484] T 5f2b92ee1a1c46bcb36bc24563886d18 P c87cbbf2d3ab4143815ef2c74ae760db reported cstate change: config changed from index 7 to 8, c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "17062e38120042aaaca05e153b2c6ccf" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { promote: false } } }
I20240501 14:16:10.351202 14418 consensus_queue.cc:237] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { promote: false } }
I20240501 14:16:10.352370 14285 raft_consensus.cc:1261] T 5f2b92ee1a1c46bcb36bc24563886d18 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Refusing update from remote peer 17062e38120042aaaca05e153b2c6ccf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:16:10.352368 15298 raft_consensus.cc:1261] T 5f2b92ee1a1c46bcb36bc24563886d18 P e3770e9581534f77a7506253b8d201b0 [term 1 FOLLOWER]: Refusing update from remote peer 17062e38120042aaaca05e153b2c6ccf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:16:10.352721 16403 consensus_queue.cc:1035] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:16:10.352880 16397 consensus_queue.cc:1035] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:16:10.354462 16468 raft_consensus.cc:2932] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { promote: false } } }
I20240501 14:16:10.354918 14285 raft_consensus.cc:2932] T 5f2b92ee1a1c46bcb36bc24563886d18 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { promote: false } } }
I20240501 14:16:10.354911 15298 raft_consensus.cc:2932] T 5f2b92ee1a1c46bcb36bc24563886d18 P e3770e9581534f77a7506253b8d201b0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { promote: false } } }
I20240501 14:16:10.355926 14129 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5f2b92ee1a1c46bcb36bc24563886d18 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:16:10.356770 14139 catalog_manager.cc:5484] T 5f2b92ee1a1c46bcb36bc24563886d18 P 17062e38120042aaaca05e153b2c6ccf reported cstate change: config changed from index 8 to 9, VOTER e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131) evicted. New cstate: current_term: 1 leader_uuid: "17062e38120042aaaca05e153b2c6ccf" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3770e9581534f77a7506253b8d201b0" member_type: VOTER last_known_addr { host: "127.7.39.132" port: 38927 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:16:10.358986 14528 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f2b92ee1a1c46bcb36bc24563886d18 with delete_type TABLET_DATA_TOMBSTONED (TS e625ab1b03d14e2aae194052d2c67bc6 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:46722
I20240501 14:16:10.359465 16484 tablet_replica.cc:330] T 5f2b92ee1a1c46bcb36bc24563886d18 P e625ab1b03d14e2aae194052d2c67bc6: stopping tablet replica
I20240501 14:16:10.359612 16484 raft_consensus.cc:2229] T 5f2b92ee1a1c46bcb36bc24563886d18 P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:16:10.359841 16484 raft_consensus.cc:2258] T 5f2b92ee1a1c46bcb36bc24563886d18 P e625ab1b03d14e2aae194052d2c67bc6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:16:10.360466 16484 ts_tablet_manager.cc:1875] T 5f2b92ee1a1c46bcb36bc24563886d18 P e625ab1b03d14e2aae194052d2c67bc6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:16:10.362295 16484 ts_tablet_manager.cc:1888] T 5f2b92ee1a1c46bcb36bc24563886d18 P e625ab1b03d14e2aae194052d2c67bc6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 14:16:10.362380 16484 log.cc:1196] T 5f2b92ee1a1c46bcb36bc24563886d18 P e625ab1b03d14e2aae194052d2c67bc6: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal/wals/5f2b92ee1a1c46bcb36bc24563886d18
I20240501 14:16:10.362854 14131 catalog_manager.cc:4830] TS e625ab1b03d14e2aae194052d2c67bc6 (127.7.39.131:33199): tablet 5f2b92ee1a1c46bcb36bc24563886d18 (table rebalance_test_table_0 [id=066c15421d0b476bbe4204501666d610]) successfully deleted
I20240501 14:16:10.378505 16450 raft_consensus.cc:1050] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6: attempting to promote NON_VOTER 17062e38120042aaaca05e153b2c6ccf to VOTER
I20240501 14:16:10.378857 16450 consensus_queue.cc:237] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { promote: false } }
I20240501 14:16:10.380137 14418 raft_consensus.cc:1261] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 17062e38120042aaaca05e153b2c6ccf [term 1 LEARNER]: Refusing update from remote peer e625ab1b03d14e2aae194052d2c67bc6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:16:10.380178 14285 raft_consensus.cc:1261] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Refusing update from remote peer e625ab1b03d14e2aae194052d2c67bc6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:16:10.380229 15168 raft_consensus.cc:1261] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 50161459714942cb9b4ae7743f76d07d [term 1 FOLLOWER]: Refusing update from remote peer e625ab1b03d14e2aae194052d2c67bc6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:16:10.380551 16461 consensus_queue.cc:1035] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:16:10.380728 16450 consensus_queue.cc:1035] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:16:10.380949 16451 consensus_queue.cc:1035] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:16:10.382895 16461 raft_consensus.cc:2932] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 17062e38120042aaaca05e153b2c6ccf (127.7.39.130) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { promote: false } } }
I20240501 14:16:10.383364 14285 raft_consensus.cc:2932] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 17062e38120042aaaca05e153b2c6ccf (127.7.39.130) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { promote: false } } }
I20240501 14:16:10.383718 15168 raft_consensus.cc:2932] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 50161459714942cb9b4ae7743f76d07d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 17062e38120042aaaca05e153b2c6ccf (127.7.39.130) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { promote: false } } }
I20240501 14:16:10.384979 14418 raft_consensus.cc:2932] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 17062e38120042aaaca05e153b2c6ccf (127.7.39.130) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { promote: false } } }
I20240501 14:16:10.386157 14141 catalog_manager.cc:5484] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 50161459714942cb9b4ae7743f76d07d reported cstate change: config changed from index 7 to 8, 17062e38120042aaaca05e153b2c6ccf (127.7.39.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e625ab1b03d14e2aae194052d2c67bc6" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c87cbbf2d3ab4143815ef2c74ae760db" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 42369 } attrs { replace: true } } peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { promote: false } } }
I20240501 14:16:10.389389 14548 consensus_queue.cc:237] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { promote: false } }
I20240501 14:16:10.390452 15168 raft_consensus.cc:1261] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 50161459714942cb9b4ae7743f76d07d [term 1 FOLLOWER]: Refusing update from remote peer e625ab1b03d14e2aae194052d2c67bc6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:16:10.390575 14418 raft_consensus.cc:1261] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Refusing update from remote peer e625ab1b03d14e2aae194052d2c67bc6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:16:10.390826 16396 consensus_queue.cc:1035] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:16:10.391018 16450 consensus_queue.cc:1035] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:16:10.392351 16396 raft_consensus.cc:2932] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P e625ab1b03d14e2aae194052d2c67bc6 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { promote: false } } }
I20240501 14:16:10.392767 15168 raft_consensus.cc:2932] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 50161459714942cb9b4ae7743f76d07d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { promote: false } } }
I20240501 14:16:10.392938 14418 raft_consensus.cc:2932] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 17062e38120042aaaca05e153b2c6ccf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { promote: false } } }
I20240501 14:16:10.394423 14131 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 47ef7ee4ea3b46cbbac2126ceaa5fa24 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:16:10.395107 14136 catalog_manager.cc:5484] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P 50161459714942cb9b4ae7743f76d07d reported cstate change: config changed from index 8 to 9, VOTER c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129) evicted. New cstate: current_term: 1 leader_uuid: "e625ab1b03d14e2aae194052d2c67bc6" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e625ab1b03d14e2aae194052d2c67bc6" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 33199 } } peers { permanent_uuid: "50161459714942cb9b4ae7743f76d07d" member_type: VOTER last_known_addr { host: "127.7.39.133" port: 36729 } attrs { promote: false } } peers { permanent_uuid: "17062e38120042aaaca05e153b2c6ccf" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 43603 } attrs { promote: false } } }
I20240501 14:16:10.397114 14264 tablet_service.cc:1514] Processing DeleteTablet for tablet 47ef7ee4ea3b46cbbac2126ceaa5fa24 with delete_type TABLET_DATA_TOMBSTONED (TS c87cbbf2d3ab4143815ef2c74ae760db not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:51888
I20240501 14:16:10.397337 16479 tablet_replica.cc:330] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P c87cbbf2d3ab4143815ef2c74ae760db: stopping tablet replica
I20240501 14:16:10.397471 16479 raft_consensus.cc:2229] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:16:10.397667 16479 raft_consensus.cc:2258] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P c87cbbf2d3ab4143815ef2c74ae760db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:16:10.398347 16479 ts_tablet_manager.cc:1875] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P c87cbbf2d3ab4143815ef2c74ae760db: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:16:10.399986 16479 ts_tablet_manager.cc:1888] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P c87cbbf2d3ab4143815ef2c74ae760db: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 14:16:10.400053 16479 log.cc:1196] T 47ef7ee4ea3b46cbbac2126ceaa5fa24 P c87cbbf2d3ab4143815ef2c74ae760db: Deleting WAL directory at /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal/wals/47ef7ee4ea3b46cbbac2126ceaa5fa24
I20240501 14:16:10.400537 14128 catalog_manager.cc:4830] TS c87cbbf2d3ab4143815ef2c74ae760db (127.7.39.129:42369): tablet 47ef7ee4ea3b46cbbac2126ceaa5fa24 (table rebalance_test_table_2 [id=1176c470113c40c796085916c4a929d7]) successfully deleted
I20240501 14:16:10.544420 14397 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:16:10.545200 14264 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:16:10.545774 15148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:16:10.550814 14528 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:16:10.552873 15279 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:16:10.628429 14397 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:16:10.629204 14264 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:16:10.630537 15148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:16:10.639667 15279 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:16:10.647398 14528 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:16:10.676240 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 14204
I20240501 14:16:10.683233 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 14335
I20240501 14:16:10.690229 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 14468
I20240501 14:16:10.696816 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 15217
I20240501 14:16:10.702204 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 15086
I20240501 14:16:10.708405 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 14113
2024-05-01T14:16:10Z chronyd exiting
[ OK ] LocationAwareRebalancingParamTest.Rebalance/idx2_rf3_t3_l_3_2 (14391 ms)
[ RUN ] LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer/idx1_rf3_t3_l_3_1
I20240501 14:16:10.736968 7326 test_util.cc:268] Using random seed: 551337284
I20240501 14:16:10.737709 7326 ts_itest-base.cc:112] Starting cluster with:
I20240501 14:16:10.737758 7326 ts_itest-base.cc:113] --------------
I20240501 14:16:10.737774 7326 ts_itest-base.cc:114] 4 tablet servers
I20240501 14:16:10.737790 7326 ts_itest-base.cc:115] 3 replicas per TS
I20240501 14:16:10.737804 7326 ts_itest-base.cc:116] --------------
2024-05-01T14:16:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T14:16:10Z Disabled control of system clock
I20240501 14:16:10.746152 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.39.190:40389
--webserver_interface=127.7.39.190
--webserver_port=0
--builtin_ntp_servers=127.7.39.148:39707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.39.190:40389
--location_mapping_cmd=/tmp/dist-test-taskhUqsdZ/build/debug/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:1
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240501 14:16:10.846094 16575 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240501 14:16:10.846643 16575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:16:10.846781 16575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:16:10.846874 16575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:16:10.849829 16575 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:16:10.849911 16575 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 14:16:10.849961 16575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:16:10.849997 16575 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 14:16:10.850031 16575 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 14:16:10.853585 16575 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:39707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskhUqsdZ/build/debug/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:1
--ipki_ca_key_size=768
--master_addresses=127.7.39.190:40389
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.39.190:40389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.7.39.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.16575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:16:10.854388 16575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:16:10.855455 16575 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:16:10.863026 16581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:16:10.863338 16580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:16:10.863572 16583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:16:10.864274 16575 server_base.cc:1034] running on GCE node
I20240501 14:16:10.864723 16575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:16:10.865590 16575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:16:10.866782 16575 hybrid_clock.cc:648] HybridClock initialized: now 1714572970866765 us; error 33 us; skew 500 ppm
I20240501 14:16:10.868656 16575 webserver.cc:460] Webserver started at http://127.7.39.190:42671/ using document root <none> and password file <none>
I20240501 14:16:10.869179 16575 fs_manager.cc:359] Metadata directory not provided
I20240501 14:16:10.869236 16575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:16:10.869417 16575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:16:10.870999 16575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "cd5a4297f1684cf6aec1692b5e89f3df"
format_stamp: "Formatted at 2024-05-01 14:16:10 on dist-test-slave-rnfw"
I20240501 14:16:10.871511 16575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "cd5a4297f1684cf6aec1692b5e89f3df"
format_stamp: "Formatted at 2024-05-01 14:16:10 on dist-test-slave-rnfw"
I20240501 14:16:10.874827 16575 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:16:10.876868 16589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:16:10.877792 16575 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:16:10.877895 16575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/wal
uuid: "cd5a4297f1684cf6aec1692b5e89f3df"
format_stamp: "Formatted at 2024-05-01 14:16:10 on dist-test-slave-rnfw"
I20240501 14:16:10.877969 16575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:16:10.899865 16575 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:16:10.900393 16575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:16:10.900539 16575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:16:10.907019 16575 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.190:40389
I20240501 14:16:10.907033 16641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.190:40389 every 8 connection(s)
I20240501 14:16:10.907837 16575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 14:16:10.910002 16642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:10.910741 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 16575
I20240501 14:16:10.910887 7326 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 14:16:10.914757 16642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd5a4297f1684cf6aec1692b5e89f3df: Bootstrap starting.
I20240501 14:16:10.916872 16642 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd5a4297f1684cf6aec1692b5e89f3df: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:10.917563 16642 log.cc:824] T 00000000000000000000000000000000 P cd5a4297f1684cf6aec1692b5e89f3df: Log is configured to *not* fsync() on all Append() calls
I20240501 14:16:10.919160 16642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd5a4297f1684cf6aec1692b5e89f3df: No bootstrap required, opened a new log
I20240501 14:16:10.921392 16642 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd5a4297f1684cf6aec1692b5e89f3df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd5a4297f1684cf6aec1692b5e89f3df" member_type: VOTER last_known_addr { host: "127.7.39.190" port: 40389 } }
I20240501 14:16:10.921543 16642 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd5a4297f1684cf6aec1692b5e89f3df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:10.921603 16642 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd5a4297f1684cf6aec1692b5e89f3df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd5a4297f1684cf6aec1692b5e89f3df, State: Initialized, Role: FOLLOWER
I20240501 14:16:10.921932 16642 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd5a4297f1684cf6aec1692b5e89f3df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd5a4297f1684cf6aec1692b5e89f3df" member_type: VOTER last_known_addr { host: "127.7.39.190" port: 40389 } }
I20240501 14:16:10.922046 16642 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd5a4297f1684cf6aec1692b5e89f3df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:16:10.922102 16642 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd5a4297f1684cf6aec1692b5e89f3df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:16:10.922197 16642 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd5a4297f1684cf6aec1692b5e89f3df [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:10.923079 16642 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd5a4297f1684cf6aec1692b5e89f3df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd5a4297f1684cf6aec1692b5e89f3df" member_type: VOTER last_known_addr { host: "127.7.39.190" port: 40389 } }
I20240501 14:16:10.923380 16642 leader_election.cc:302] T 00000000000000000000000000000000 P cd5a4297f1684cf6aec1692b5e89f3df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd5a4297f1684cf6aec1692b5e89f3df; no voters:
I20240501 14:16:10.923635 16642 leader_election.cc:288] T 00000000000000000000000000000000 P cd5a4297f1684cf6aec1692b5e89f3df [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:16:10.923738 16647 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd5a4297f1684cf6aec1692b5e89f3df [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:16:10.923976 16647 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd5a4297f1684cf6aec1692b5e89f3df [term 1 LEADER]: Becoming Leader. State: Replica: cd5a4297f1684cf6aec1692b5e89f3df, State: Running, Role: LEADER
I20240501 14:16:10.924311 16647 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd5a4297f1684cf6aec1692b5e89f3df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd5a4297f1684cf6aec1692b5e89f3df" member_type: VOTER last_known_addr { host: "127.7.39.190" port: 40389 } }
I20240501 14:16:10.924659 16642 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd5a4297f1684cf6aec1692b5e89f3df [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:16:10.925602 16648 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd5a4297f1684cf6aec1692b5e89f3df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd5a4297f1684cf6aec1692b5e89f3df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd5a4297f1684cf6aec1692b5e89f3df" member_type: VOTER last_known_addr { host: "127.7.39.190" port: 40389 } } }
I20240501 14:16:10.925714 16648 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd5a4297f1684cf6aec1692b5e89f3df [sys.catalog]: This master's current role is: LEADER
I20240501 14:16:10.925979 16649 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd5a4297f1684cf6aec1692b5e89f3df [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd5a4297f1684cf6aec1692b5e89f3df. Latest consensus state: current_term: 1 leader_uuid: "cd5a4297f1684cf6aec1692b5e89f3df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd5a4297f1684cf6aec1692b5e89f3df" member_type: VOTER last_known_addr { host: "127.7.39.190" port: 40389 } } }
I20240501 14:16:10.926121 16649 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd5a4297f1684cf6aec1692b5e89f3df [sys.catalog]: This master's current role is: LEADER
I20240501 14:16:10.926131 16656 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:16:10.928795 16656 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:16:10.933233 16656 catalog_manager.cc:1347] Generated new cluster ID: 2b7b81ef0bf74352be7e163832ac682e
I20240501 14:16:10.933293 16656 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:16:10.941810 16656 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 14:16:10.942566 16656 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:16:10.948621 16656 catalog_manager.cc:5857] T 00000000000000000000000000000000 P cd5a4297f1684cf6aec1692b5e89f3df: Generated new TSK 0
I20240501 14:16:10.949113 16656 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:16:10.959206 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.129:0
--local_ip_for_outbound_sockets=127.7.39.129
--webserver_interface=127.7.39.129
--webserver_port=0
--tserver_master_addrs=127.7.39.190:40389
--builtin_ntp_servers=127.7.39.148:39707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 14:16:11.062114 16666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:16:11.062316 16666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:16:11.062388 16666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:16:11.065351 16666 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:16:11.065436 16666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:16:11.065523 16666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.129
I20240501 14:16:11.068933 16666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:39707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.7.39.129
--webserver_port=0
--tserver_master_addrs=127.7.39.190:40389
--never_fsync=true
--heap_profile_path=/tmp/kudu.16666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.129
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:16:11.069717 16666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:16:11.070760 16666 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:16:11.078270 16671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:16:11.078752 16672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:16:11.079380 16674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:16:11.080039 16666 server_base.cc:1034] running on GCE node
I20240501 14:16:11.080406 16666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:16:11.080942 16666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:16:11.082098 16666 hybrid_clock.cc:648] HybridClock initialized: now 1714572971082086 us; error 31 us; skew 500 ppm
I20240501 14:16:11.084025 16666 webserver.cc:460] Webserver started at http://127.7.39.129:43429/ using document root <none> and password file <none>
I20240501 14:16:11.084542 16666 fs_manager.cc:359] Metadata directory not provided
I20240501 14:16:11.084595 16666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:16:11.084754 16666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:16:11.086261 16666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "626e808943c54503bc4bd8cf1cd3e4b5"
format_stamp: "Formatted at 2024-05-01 14:16:11 on dist-test-slave-rnfw"
I20240501 14:16:11.086722 16666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "626e808943c54503bc4bd8cf1cd3e4b5"
format_stamp: "Formatted at 2024-05-01 14:16:11 on dist-test-slave-rnfw"
I20240501 14:16:11.089745 16666 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 14:16:11.091714 16680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:16:11.092633 16666 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:16:11.092728 16666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "626e808943c54503bc4bd8cf1cd3e4b5"
format_stamp: "Formatted at 2024-05-01 14:16:11 on dist-test-slave-rnfw"
I20240501 14:16:11.092811 16666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:16:11.116520 16666 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:16:11.116992 16666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:16:11.117130 16666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:16:11.117643 16666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:16:11.118469 16666 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:16:11.118515 16666 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:16:11.118551 16666 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:16:11.118587 16666 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:16:11.127018 16666 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.129:43915
I20240501 14:16:11.127089 16793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.129:43915 every 8 connection(s)
I20240501 14:16:11.127815 16666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 14:16:11.135735 16794 heartbeater.cc:347] Connected to a master server at 127.7.39.190:40389
I20240501 14:16:11.135910 16794 heartbeater.cc:464] Registering TS with master...
I20240501 14:16:11.136325 16794 heartbeater.cc:511] Master 127.7.39.190:40389 requested a full tablet report, sending...
I20240501 14:16:11.136746 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 16666
I20240501 14:16:11.136864 7326 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 14:16:11.141307 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.130:0
--local_ip_for_outbound_sockets=127.7.39.130
--webserver_interface=127.7.39.130
--webserver_port=0
--tserver_master_addrs=127.7.39.190:40389
--builtin_ntp_servers=127.7.39.148:39707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:16:11.172188 16606 ts_manager.cc:194] Registered new tserver with Master: 626e808943c54503bc4bd8cf1cd3e4b5 (127.7.39.129:43915)
I20240501 14:16:11.173514 16606 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.129:43313
W20240501 14:16:11.244057 16798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:16:11.244238 16798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:16:11.244279 16798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:16:11.247113 16798 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:16:11.247166 16798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:16:11.247251 16798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.130
I20240501 14:16:11.250593 16798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:39707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.7.39.130
--webserver_port=0
--tserver_master_addrs=127.7.39.190:40389
--never_fsync=true
--heap_profile_path=/tmp/kudu.16798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.130
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:16:11.251287 16798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:16:11.252270 16798 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:16:11.259447 16806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:16:11.260020 16803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:16:11.260615 16804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:16:11.261265 16798 server_base.cc:1034] running on GCE node
I20240501 14:16:11.261557 16798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:16:11.262005 16798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:16:11.263109 16798 hybrid_clock.cc:648] HybridClock initialized: now 1714572971263087 us; error 42 us; skew 500 ppm
I20240501 14:16:11.264981 16798 webserver.cc:460] Webserver started at http://127.7.39.130:35041/ using document root <none> and password file <none>
I20240501 14:16:11.265461 16798 fs_manager.cc:359] Metadata directory not provided
I20240501 14:16:11.265517 16798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:16:11.265669 16798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:16:11.267104 16798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "e1a9c39d088a43e7b6f9beee2068deeb"
format_stamp: "Formatted at 2024-05-01 14:16:11 on dist-test-slave-rnfw"
I20240501 14:16:11.267580 16798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "e1a9c39d088a43e7b6f9beee2068deeb"
format_stamp: "Formatted at 2024-05-01 14:16:11 on dist-test-slave-rnfw"
I20240501 14:16:11.270777 16798 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:16:11.272904 16812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:16:11.273876 16798 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:16:11.273989 16798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "e1a9c39d088a43e7b6f9beee2068deeb"
format_stamp: "Formatted at 2024-05-01 14:16:11 on dist-test-slave-rnfw"
I20240501 14:16:11.274068 16798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:16:11.288081 16798 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:16:11.288538 16798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:16:11.288676 16798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:16:11.289125 16798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:16:11.289877 16798 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:16:11.289932 16798 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:16:11.289983 16798 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:16:11.290006 16798 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:16:11.297890 16798 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.130:40479
I20240501 14:16:11.297912 16925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.130:40479 every 8 connection(s)
I20240501 14:16:11.298641 16798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 14:16:11.306254 16926 heartbeater.cc:347] Connected to a master server at 127.7.39.190:40389
I20240501 14:16:11.306455 16926 heartbeater.cc:464] Registering TS with master...
I20240501 14:16:11.306916 16926 heartbeater.cc:511] Master 127.7.39.190:40389 requested a full tablet report, sending...
I20240501 14:16:11.307178 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 16798
I20240501 14:16:11.307301 7326 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 14:16:11.311728 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.131:0
--local_ip_for_outbound_sockets=127.7.39.131
--webserver_interface=127.7.39.131
--webserver_port=0
--tserver_master_addrs=127.7.39.190:40389
--builtin_ntp_servers=127.7.39.148:39707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:16:11.340495 16606 ts_manager.cc:194] Registered new tserver with Master: e1a9c39d088a43e7b6f9beee2068deeb (127.7.39.130:40479)
I20240501 14:16:11.341398 16606 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.130:50825
W20240501 14:16:11.408866 16930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:16:11.409044 16930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:16:11.409083 16930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:16:11.411880 16930 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:16:11.411937 16930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:16:11.411999 16930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.131
I20240501 14:16:11.415213 16930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:39707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.7.39.131
--webserver_port=0
--tserver_master_addrs=127.7.39.190:40389
--never_fsync=true
--heap_profile_path=/tmp/kudu.16930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.131
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:16:11.415853 16930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:16:11.416760 16930 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:16:11.423907 16935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:16:11.424490 16936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:16:11.425124 16938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:16:11.425748 16930 server_base.cc:1034] running on GCE node
I20240501 14:16:11.426028 16930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:16:11.426539 16930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:16:11.427702 16930 hybrid_clock.cc:648] HybridClock initialized: now 1714572971427674 us; error 42 us; skew 500 ppm
I20240501 14:16:11.429351 16930 webserver.cc:460] Webserver started at http://127.7.39.131:40625/ using document root <none> and password file <none>
I20240501 14:16:11.429888 16930 fs_manager.cc:359] Metadata directory not provided
I20240501 14:16:11.429953 16930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:16:11.430094 16930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:16:11.431519 16930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "48789b208dcf4f8ab388470fa43a751b"
format_stamp: "Formatted at 2024-05-01 14:16:11 on dist-test-slave-rnfw"
I20240501 14:16:11.431892 16930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "48789b208dcf4f8ab388470fa43a751b"
format_stamp: "Formatted at 2024-05-01 14:16:11 on dist-test-slave-rnfw"
I20240501 14:16:11.434901 16930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:16:11.436722 16944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:16:11.437577 16930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 14:16:11.437667 16930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "48789b208dcf4f8ab388470fa43a751b"
format_stamp: "Formatted at 2024-05-01 14:16:11 on dist-test-slave-rnfw"
I20240501 14:16:11.437733 16930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:16:11.447953 16930 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:16:11.448438 16930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:16:11.448565 16930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:16:11.448982 16930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:16:11.449700 16930 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:16:11.449743 16930 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:16:11.449793 16930 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:16:11.449820 16930 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:16:11.457854 16930 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.131:45011
I20240501 14:16:11.457901 17057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.131:45011 every 8 connection(s)
I20240501 14:16:11.458597 16930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 14:16:11.466202 17058 heartbeater.cc:347] Connected to a master server at 127.7.39.190:40389
I20240501 14:16:11.466382 17058 heartbeater.cc:464] Registering TS with master...
I20240501 14:16:11.466804 17058 heartbeater.cc:511] Master 127.7.39.190:40389 requested a full tablet report, sending...
I20240501 14:16:11.467113 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 16930
I20240501 14:16:11.467211 7326 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 14:16:11.471432 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.132:0
--local_ip_for_outbound_sockets=127.7.39.132
--webserver_interface=127.7.39.132
--webserver_port=0
--tserver_master_addrs=127.7.39.190:40389
--builtin_ntp_servers=127.7.39.148:39707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:16:11.498366 16606 ts_manager.cc:194] Registered new tserver with Master: 48789b208dcf4f8ab388470fa43a751b (127.7.39.131:45011)
I20240501 14:16:11.498988 16606 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.131:60729
W20240501 14:16:11.567996 17062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:16:11.568153 17062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:16:11.568192 17062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:16:11.570947 17062 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:16:11.571000 17062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:16:11.571101 17062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.132
I20240501 14:16:11.574369 17062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:39707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.7.39.132
--webserver_port=0
--tserver_master_addrs=127.7.39.190:40389
--never_fsync=true
--heap_profile_path=/tmp/kudu.17062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.132
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:16:11.575039 17062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:16:11.576061 17062 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:16:11.583891 17067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:16:11.584322 17070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:16:11.584599 17068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:16:11.585369 17062 server_base.cc:1034] running on GCE node
I20240501 14:16:11.585676 17062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:16:11.586119 17062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:16:11.587265 17062 hybrid_clock.cc:648] HybridClock initialized: now 1714572971587241 us; error 43 us; skew 500 ppm
I20240501 14:16:11.589005 17062 webserver.cc:460] Webserver started at http://127.7.39.132:41037/ using document root <none> and password file <none>
I20240501 14:16:11.589475 17062 fs_manager.cc:359] Metadata directory not provided
I20240501 14:16:11.589540 17062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:16:11.589695 17062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:16:11.591178 17062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "5852487cc13a4313b0a727f9eb1a6d6a"
format_stamp: "Formatted at 2024-05-01 14:16:11 on dist-test-slave-rnfw"
I20240501 14:16:11.591653 17062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "5852487cc13a4313b0a727f9eb1a6d6a"
format_stamp: "Formatted at 2024-05-01 14:16:11 on dist-test-slave-rnfw"
I20240501 14:16:11.594583 17062 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240501 14:16:11.596571 17076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:16:11.597481 17062 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240501 14:16:11.597568 17062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "5852487cc13a4313b0a727f9eb1a6d6a"
format_stamp: "Formatted at 2024-05-01 14:16:11 on dist-test-slave-rnfw"
I20240501 14:16:11.597631 17062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:16:11.608640 17062 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:16:11.609062 17062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:16:11.609179 17062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:16:11.609577 17062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:16:11.610277 17062 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:16:11.610322 17062 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:16:11.610356 17062 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:16:11.610370 17062 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:16:11.618191 17062 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.132:40745
I20240501 14:16:11.618212 17189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.132:40745 every 8 connection(s)
I20240501 14:16:11.618966 17062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 14:16:11.626740 17190 heartbeater.cc:347] Connected to a master server at 127.7.39.190:40389
I20240501 14:16:11.626775 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 17062
I20240501 14:16:11.626899 7326 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 14:16:11.626917 17190 heartbeater.cc:464] Registering TS with master...
I20240501 14:16:11.627368 17190 heartbeater.cc:511] Master 127.7.39.190:40389 requested a full tablet report, sending...
I20240501 14:16:11.660523 16606 ts_manager.cc:194] Registered new tserver with Master: 5852487cc13a4313b0a727f9eb1a6d6a (127.7.39.132:40745)
I20240501 14:16:11.661135 16606 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.132:36409
I20240501 14:16:11.666208 7326 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240501 14:16:11.674729 7326 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240501 14:16:11.674789 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 17062
I20240501 14:16:12.176057 16794 heartbeater.cc:503] Master 127.7.39.190:40389 was elected leader, sending a full tablet report...
I20240501 14:16:12.343564 16926 heartbeater.cc:503] Master 127.7.39.190:40389 was elected leader, sending a full tablet report...
I20240501 14:16:12.500867 17058 heartbeater.cc:503] Master 127.7.39.190:40389 was elected leader, sending a full tablet report...
I20240501 14:16:15.432030 16606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36462:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240501 14:16:15.435168 16606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 14:16:15.452314 16859 tablet_service.cc:1467] Processing CreateTablet for tablet 5554988a4ab04a51abb784a41999e1a4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=699b6d4609d44b40906b3e01f858ba63]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:15.453303 16859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5554988a4ab04a51abb784a41999e1a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:15.453438 16991 tablet_service.cc:1467] Processing CreateTablet for tablet 5554988a4ab04a51abb784a41999e1a4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=699b6d4609d44b40906b3e01f858ba63]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:15.453188 16990 tablet_service.cc:1467] Processing CreateTablet for tablet 75227f95427f4c6d8dc2b31b5f4072e9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=699b6d4609d44b40906b3e01f858ba63]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:15.453171 16992 tablet_service.cc:1467] Processing CreateTablet for tablet cbcda590e51c480ba98ddaf9370c34fa (DEFAULT_TABLE table=rebalance_test_table_0 [id=699b6d4609d44b40906b3e01f858ba63]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:15.454140 16990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75227f95427f4c6d8dc2b31b5f4072e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:15.452314 16858 tablet_service.cc:1467] Processing CreateTablet for tablet 75227f95427f4c6d8dc2b31b5f4072e9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=699b6d4609d44b40906b3e01f858ba63]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:15.454998 16726 tablet_service.cc:1467] Processing CreateTablet for tablet 75227f95427f4c6d8dc2b31b5f4072e9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=699b6d4609d44b40906b3e01f858ba63]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:15.455437 16991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5554988a4ab04a51abb784a41999e1a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:15.455503 16858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75227f95427f4c6d8dc2b31b5f4072e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:15.454998 16728 tablet_service.cc:1467] Processing CreateTablet for tablet cbcda590e51c480ba98ddaf9370c34fa (DEFAULT_TABLE table=rebalance_test_table_0 [id=699b6d4609d44b40906b3e01f858ba63]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:15.455159 16727 tablet_service.cc:1467] Processing CreateTablet for tablet 5554988a4ab04a51abb784a41999e1a4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=699b6d4609d44b40906b3e01f858ba63]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:15.455852 16726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75227f95427f4c6d8dc2b31b5f4072e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:15.456046 16992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbcda590e51c480ba98ddaf9370c34fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:15.452315 16860 tablet_service.cc:1467] Processing CreateTablet for tablet cbcda590e51c480ba98ddaf9370c34fa (DEFAULT_TABLE table=rebalance_test_table_0 [id=699b6d4609d44b40906b3e01f858ba63]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:15.456490 16860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbcda590e51c480ba98ddaf9370c34fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:15.461854 16727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5554988a4ab04a51abb784a41999e1a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:15.463436 17211 tablet_bootstrap.cc:492] T 75227f95427f4c6d8dc2b31b5f4072e9 P e1a9c39d088a43e7b6f9beee2068deeb: Bootstrap starting.
I20240501 14:16:15.464141 16728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbcda590e51c480ba98ddaf9370c34fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:15.465876 17211 tablet_bootstrap.cc:654] T 75227f95427f4c6d8dc2b31b5f4072e9 P e1a9c39d088a43e7b6f9beee2068deeb: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:15.466295 17209 tablet_bootstrap.cc:492] T 5554988a4ab04a51abb784a41999e1a4 P 48789b208dcf4f8ab388470fa43a751b: Bootstrap starting.
I20240501 14:16:15.466661 17211 log.cc:824] T 75227f95427f4c6d8dc2b31b5f4072e9 P e1a9c39d088a43e7b6f9beee2068deeb: Log is configured to *not* fsync() on all Append() calls
I20240501 14:16:15.467020 17210 tablet_bootstrap.cc:492] T 75227f95427f4c6d8dc2b31b5f4072e9 P 626e808943c54503bc4bd8cf1cd3e4b5: Bootstrap starting.
I20240501 14:16:15.467862 17209 tablet_bootstrap.cc:654] T 5554988a4ab04a51abb784a41999e1a4 P 48789b208dcf4f8ab388470fa43a751b: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:15.468426 17209 log.cc:824] T 5554988a4ab04a51abb784a41999e1a4 P 48789b208dcf4f8ab388470fa43a751b: Log is configured to *not* fsync() on all Append() calls
I20240501 14:16:15.468583 17210 tablet_bootstrap.cc:654] T 75227f95427f4c6d8dc2b31b5f4072e9 P 626e808943c54503bc4bd8cf1cd3e4b5: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:15.469291 17210 log.cc:824] T 75227f95427f4c6d8dc2b31b5f4072e9 P 626e808943c54503bc4bd8cf1cd3e4b5: Log is configured to *not* fsync() on all Append() calls
I20240501 14:16:15.470144 17209 tablet_bootstrap.cc:492] T 5554988a4ab04a51abb784a41999e1a4 P 48789b208dcf4f8ab388470fa43a751b: No bootstrap required, opened a new log
I20240501 14:16:15.470242 17211 tablet_bootstrap.cc:492] T 75227f95427f4c6d8dc2b31b5f4072e9 P e1a9c39d088a43e7b6f9beee2068deeb: No bootstrap required, opened a new log
I20240501 14:16:15.470381 17209 ts_tablet_manager.cc:1367] T 5554988a4ab04a51abb784a41999e1a4 P 48789b208dcf4f8ab388470fa43a751b: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240501 14:16:15.470450 17211 ts_tablet_manager.cc:1367] T 75227f95427f4c6d8dc2b31b5f4072e9 P e1a9c39d088a43e7b6f9beee2068deeb: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240501 14:16:15.471146 17210 tablet_bootstrap.cc:492] T 75227f95427f4c6d8dc2b31b5f4072e9 P 626e808943c54503bc4bd8cf1cd3e4b5: No bootstrap required, opened a new log
I20240501 14:16:15.471326 17210 ts_tablet_manager.cc:1367] T 75227f95427f4c6d8dc2b31b5f4072e9 P 626e808943c54503bc4bd8cf1cd3e4b5: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20240501 14:16:15.473080 17211 raft_consensus.cc:348] T 75227f95427f4c6d8dc2b31b5f4072e9 P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:15.473256 17211 raft_consensus.cc:374] T 75227f95427f4c6d8dc2b31b5f4072e9 P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:15.473330 17211 raft_consensus.cc:729] T 75227f95427f4c6d8dc2b31b5f4072e9 P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a9c39d088a43e7b6f9beee2068deeb, State: Initialized, Role: FOLLOWER
I20240501 14:16:15.473753 17211 consensus_queue.cc:260] T 75227f95427f4c6d8dc2b31b5f4072e9 P e1a9c39d088a43e7b6f9beee2068deeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:15.473871 17209 raft_consensus.cc:348] T 5554988a4ab04a51abb784a41999e1a4 P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:15.474129 17209 raft_consensus.cc:374] T 5554988a4ab04a51abb784a41999e1a4 P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:15.474220 17209 raft_consensus.cc:729] T 5554988a4ab04a51abb784a41999e1a4 P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48789b208dcf4f8ab388470fa43a751b, State: Initialized, Role: FOLLOWER
I20240501 14:16:15.474617 17211 ts_tablet_manager.cc:1398] T 75227f95427f4c6d8dc2b31b5f4072e9 P e1a9c39d088a43e7b6f9beee2068deeb: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 14:16:15.474644 17209 consensus_queue.cc:260] T 5554988a4ab04a51abb784a41999e1a4 P 48789b208dcf4f8ab388470fa43a751b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:15.474946 17211 tablet_bootstrap.cc:492] T cbcda590e51c480ba98ddaf9370c34fa P e1a9c39d088a43e7b6f9beee2068deeb: Bootstrap starting.
I20240501 14:16:15.475430 17209 ts_tablet_manager.cc:1398] T 5554988a4ab04a51abb784a41999e1a4 P 48789b208dcf4f8ab388470fa43a751b: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240501 14:16:15.475981 17211 tablet_bootstrap.cc:654] T cbcda590e51c480ba98ddaf9370c34fa P e1a9c39d088a43e7b6f9beee2068deeb: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:15.476168 17209 tablet_bootstrap.cc:492] T 75227f95427f4c6d8dc2b31b5f4072e9 P 48789b208dcf4f8ab388470fa43a751b: Bootstrap starting.
I20240501 14:16:15.477154 17209 tablet_bootstrap.cc:654] T 75227f95427f4c6d8dc2b31b5f4072e9 P 48789b208dcf4f8ab388470fa43a751b: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:15.477167 17211 tablet_bootstrap.cc:492] T cbcda590e51c480ba98ddaf9370c34fa P e1a9c39d088a43e7b6f9beee2068deeb: No bootstrap required, opened a new log
I20240501 14:16:15.477241 17211 ts_tablet_manager.cc:1367] T cbcda590e51c480ba98ddaf9370c34fa P e1a9c39d088a43e7b6f9beee2068deeb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:16:15.477370 17210 raft_consensus.cc:348] T 75227f95427f4c6d8dc2b31b5f4072e9 P 626e808943c54503bc4bd8cf1cd3e4b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:15.477589 17210 raft_consensus.cc:374] T 75227f95427f4c6d8dc2b31b5f4072e9 P 626e808943c54503bc4bd8cf1cd3e4b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:15.477675 17210 raft_consensus.cc:729] T 75227f95427f4c6d8dc2b31b5f4072e9 P 626e808943c54503bc4bd8cf1cd3e4b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 626e808943c54503bc4bd8cf1cd3e4b5, State: Initialized, Role: FOLLOWER
I20240501 14:16:15.477852 17211 raft_consensus.cc:348] T cbcda590e51c480ba98ddaf9370c34fa P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:15.478039 17211 raft_consensus.cc:374] T cbcda590e51c480ba98ddaf9370c34fa P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:15.478102 17211 raft_consensus.cc:729] T cbcda590e51c480ba98ddaf9370c34fa P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a9c39d088a43e7b6f9beee2068deeb, State: Initialized, Role: FOLLOWER
I20240501 14:16:15.478098 17210 consensus_queue.cc:260] T 75227f95427f4c6d8dc2b31b5f4072e9 P 626e808943c54503bc4bd8cf1cd3e4b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:15.478257 17211 consensus_queue.cc:260] T cbcda590e51c480ba98ddaf9370c34fa P e1a9c39d088a43e7b6f9beee2068deeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:15.478475 17209 tablet_bootstrap.cc:492] T 75227f95427f4c6d8dc2b31b5f4072e9 P 48789b208dcf4f8ab388470fa43a751b: No bootstrap required, opened a new log
I20240501 14:16:15.478582 17209 ts_tablet_manager.cc:1367] T 75227f95427f4c6d8dc2b31b5f4072e9 P 48789b208dcf4f8ab388470fa43a751b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:16:15.478603 17211 ts_tablet_manager.cc:1398] T cbcda590e51c480ba98ddaf9370c34fa P e1a9c39d088a43e7b6f9beee2068deeb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:16:15.478754 17211 tablet_bootstrap.cc:492] T 5554988a4ab04a51abb784a41999e1a4 P e1a9c39d088a43e7b6f9beee2068deeb: Bootstrap starting.
I20240501 14:16:15.478925 17210 ts_tablet_manager.cc:1398] T 75227f95427f4c6d8dc2b31b5f4072e9 P 626e808943c54503bc4bd8cf1cd3e4b5: Time spent starting tablet: real 0.008s user 0.003s sys 0.002s
I20240501 14:16:15.479099 17209 raft_consensus.cc:348] T 75227f95427f4c6d8dc2b31b5f4072e9 P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:15.479188 17209 raft_consensus.cc:374] T 75227f95427f4c6d8dc2b31b5f4072e9 P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:15.479221 17209 raft_consensus.cc:729] T 75227f95427f4c6d8dc2b31b5f4072e9 P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48789b208dcf4f8ab388470fa43a751b, State: Initialized, Role: FOLLOWER
I20240501 14:16:15.479280 17210 tablet_bootstrap.cc:492] T 5554988a4ab04a51abb784a41999e1a4 P 626e808943c54503bc4bd8cf1cd3e4b5: Bootstrap starting.
I20240501 14:16:15.479331 17209 consensus_queue.cc:260] T 75227f95427f4c6d8dc2b31b5f4072e9 P 48789b208dcf4f8ab388470fa43a751b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:15.479600 17209 ts_tablet_manager.cc:1398] T 75227f95427f4c6d8dc2b31b5f4072e9 P 48789b208dcf4f8ab388470fa43a751b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:16:15.479612 17211 tablet_bootstrap.cc:654] T 5554988a4ab04a51abb784a41999e1a4 P e1a9c39d088a43e7b6f9beee2068deeb: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:15.479754 17209 tablet_bootstrap.cc:492] T cbcda590e51c480ba98ddaf9370c34fa P 48789b208dcf4f8ab388470fa43a751b: Bootstrap starting.
I20240501 14:16:15.480234 17210 tablet_bootstrap.cc:654] T 5554988a4ab04a51abb784a41999e1a4 P 626e808943c54503bc4bd8cf1cd3e4b5: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:15.480651 17209 tablet_bootstrap.cc:654] T cbcda590e51c480ba98ddaf9370c34fa P 48789b208dcf4f8ab388470fa43a751b: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:15.481173 17210 tablet_bootstrap.cc:492] T 5554988a4ab04a51abb784a41999e1a4 P 626e808943c54503bc4bd8cf1cd3e4b5: No bootstrap required, opened a new log
I20240501 14:16:15.481241 17210 ts_tablet_manager.cc:1367] T 5554988a4ab04a51abb784a41999e1a4 P 626e808943c54503bc4bd8cf1cd3e4b5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:16:15.481782 17210 raft_consensus.cc:348] T 5554988a4ab04a51abb784a41999e1a4 P 626e808943c54503bc4bd8cf1cd3e4b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:15.481870 17211 tablet_bootstrap.cc:492] T 5554988a4ab04a51abb784a41999e1a4 P e1a9c39d088a43e7b6f9beee2068deeb: No bootstrap required, opened a new log
I20240501 14:16:15.481876 17210 raft_consensus.cc:374] T 5554988a4ab04a51abb784a41999e1a4 P 626e808943c54503bc4bd8cf1cd3e4b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:15.481917 17210 raft_consensus.cc:729] T 5554988a4ab04a51abb784a41999e1a4 P 626e808943c54503bc4bd8cf1cd3e4b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 626e808943c54503bc4bd8cf1cd3e4b5, State: Initialized, Role: FOLLOWER
I20240501 14:16:15.481957 17211 ts_tablet_manager.cc:1367] T 5554988a4ab04a51abb784a41999e1a4 P e1a9c39d088a43e7b6f9beee2068deeb: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 14:16:15.482041 17210 consensus_queue.cc:260] T 5554988a4ab04a51abb784a41999e1a4 P 626e808943c54503bc4bd8cf1cd3e4b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:15.482331 17210 ts_tablet_manager.cc:1398] T 5554988a4ab04a51abb784a41999e1a4 P 626e808943c54503bc4bd8cf1cd3e4b5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:16:15.482460 17211 raft_consensus.cc:348] T 5554988a4ab04a51abb784a41999e1a4 P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:15.482553 17211 raft_consensus.cc:374] T 5554988a4ab04a51abb784a41999e1a4 P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:15.482587 17211 raft_consensus.cc:729] T 5554988a4ab04a51abb784a41999e1a4 P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a9c39d088a43e7b6f9beee2068deeb, State: Initialized, Role: FOLLOWER
I20240501 14:16:15.482697 17211 consensus_queue.cc:260] T 5554988a4ab04a51abb784a41999e1a4 P e1a9c39d088a43e7b6f9beee2068deeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:15.482944 17211 ts_tablet_manager.cc:1398] T 5554988a4ab04a51abb784a41999e1a4 P e1a9c39d088a43e7b6f9beee2068deeb: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:16:15.482977 17210 tablet_bootstrap.cc:492] T cbcda590e51c480ba98ddaf9370c34fa P 626e808943c54503bc4bd8cf1cd3e4b5: Bootstrap starting.
I20240501 14:16:15.483743 17209 tablet_bootstrap.cc:492] T cbcda590e51c480ba98ddaf9370c34fa P 48789b208dcf4f8ab388470fa43a751b: No bootstrap required, opened a new log
I20240501 14:16:15.483822 17209 ts_tablet_manager.cc:1367] T cbcda590e51c480ba98ddaf9370c34fa P 48789b208dcf4f8ab388470fa43a751b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:16:15.484159 17210 tablet_bootstrap.cc:654] T cbcda590e51c480ba98ddaf9370c34fa P 626e808943c54503bc4bd8cf1cd3e4b5: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:15.484287 17209 raft_consensus.cc:348] T cbcda590e51c480ba98ddaf9370c34fa P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:15.484400 17209 raft_consensus.cc:374] T cbcda590e51c480ba98ddaf9370c34fa P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:15.484436 17209 raft_consensus.cc:729] T cbcda590e51c480ba98ddaf9370c34fa P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48789b208dcf4f8ab388470fa43a751b, State: Initialized, Role: FOLLOWER
I20240501 14:16:15.484549 17209 consensus_queue.cc:260] T cbcda590e51c480ba98ddaf9370c34fa P 48789b208dcf4f8ab388470fa43a751b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:15.484794 17209 ts_tablet_manager.cc:1398] T cbcda590e51c480ba98ddaf9370c34fa P 48789b208dcf4f8ab388470fa43a751b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:16:15.485544 17210 tablet_bootstrap.cc:492] T cbcda590e51c480ba98ddaf9370c34fa P 626e808943c54503bc4bd8cf1cd3e4b5: No bootstrap required, opened a new log
I20240501 14:16:15.485641 17210 ts_tablet_manager.cc:1367] T cbcda590e51c480ba98ddaf9370c34fa P 626e808943c54503bc4bd8cf1cd3e4b5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:16:15.486063 17210 raft_consensus.cc:348] T cbcda590e51c480ba98ddaf9370c34fa P 626e808943c54503bc4bd8cf1cd3e4b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:15.486169 17210 raft_consensus.cc:374] T cbcda590e51c480ba98ddaf9370c34fa P 626e808943c54503bc4bd8cf1cd3e4b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:15.486203 17210 raft_consensus.cc:729] T cbcda590e51c480ba98ddaf9370c34fa P 626e808943c54503bc4bd8cf1cd3e4b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 626e808943c54503bc4bd8cf1cd3e4b5, State: Initialized, Role: FOLLOWER
I20240501 14:16:15.486322 17210 consensus_queue.cc:260] T cbcda590e51c480ba98ddaf9370c34fa P 626e808943c54503bc4bd8cf1cd3e4b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:15.486537 17210 ts_tablet_manager.cc:1398] T cbcda590e51c480ba98ddaf9370c34fa P 626e808943c54503bc4bd8cf1cd3e4b5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:16:15.510731 17219 raft_consensus.cc:482] T 5554988a4ab04a51abb784a41999e1a4 P 626e808943c54503bc4bd8cf1cd3e4b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:16:15.510959 17219 raft_consensus.cc:504] T 5554988a4ab04a51abb784a41999e1a4 P 626e808943c54503bc4bd8cf1cd3e4b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:15.512236 17219 leader_election.cc:288] T 5554988a4ab04a51abb784a41999e1a4 P 626e808943c54503bc4bd8cf1cd3e4b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1a9c39d088a43e7b6f9beee2068deeb (127.7.39.130:40479), 48789b208dcf4f8ab388470fa43a751b (127.7.39.131:45011)
I20240501 14:16:15.516574 16880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5554988a4ab04a51abb784a41999e1a4" candidate_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" is_pre_election: true
I20240501 14:16:15.516582 17012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5554988a4ab04a51abb784a41999e1a4" candidate_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48789b208dcf4f8ab388470fa43a751b" is_pre_election: true
I20240501 14:16:15.516834 17012 raft_consensus.cc:2454] T 5554988a4ab04a51abb784a41999e1a4 P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 626e808943c54503bc4bd8cf1cd3e4b5 in term 0.
I20240501 14:16:15.516845 16880 raft_consensus.cc:2454] T 5554988a4ab04a51abb784a41999e1a4 P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 626e808943c54503bc4bd8cf1cd3e4b5 in term 0.
I20240501 14:16:15.517242 16684 leader_election.cc:302] T 5554988a4ab04a51abb784a41999e1a4 P 626e808943c54503bc4bd8cf1cd3e4b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 626e808943c54503bc4bd8cf1cd3e4b5, e1a9c39d088a43e7b6f9beee2068deeb; no voters:
I20240501 14:16:15.517480 17219 raft_consensus.cc:2781] T 5554988a4ab04a51abb784a41999e1a4 P 626e808943c54503bc4bd8cf1cd3e4b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:16:15.517563 17219 raft_consensus.cc:482] T 5554988a4ab04a51abb784a41999e1a4 P 626e808943c54503bc4bd8cf1cd3e4b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:16:15.517603 17219 raft_consensus.cc:3037] T 5554988a4ab04a51abb784a41999e1a4 P 626e808943c54503bc4bd8cf1cd3e4b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:15.518472 17219 raft_consensus.cc:504] T 5554988a4ab04a51abb784a41999e1a4 P 626e808943c54503bc4bd8cf1cd3e4b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:15.518797 17219 leader_election.cc:288] T 5554988a4ab04a51abb784a41999e1a4 P 626e808943c54503bc4bd8cf1cd3e4b5 [CANDIDATE]: Term 1 election: Requested vote from peers e1a9c39d088a43e7b6f9beee2068deeb (127.7.39.130:40479), 48789b208dcf4f8ab388470fa43a751b (127.7.39.131:45011)
I20240501 14:16:15.519146 16880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5554988a4ab04a51abb784a41999e1a4" candidate_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9c39d088a43e7b6f9beee2068deeb"
I20240501 14:16:15.519168 17012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5554988a4ab04a51abb784a41999e1a4" candidate_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48789b208dcf4f8ab388470fa43a751b"
I20240501 14:16:15.519256 17012 raft_consensus.cc:3037] T 5554988a4ab04a51abb784a41999e1a4 P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:15.519260 16880 raft_consensus.cc:3037] T 5554988a4ab04a51abb784a41999e1a4 P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:15.520370 17012 raft_consensus.cc:2454] T 5554988a4ab04a51abb784a41999e1a4 P 48789b208dcf4f8ab388470fa43a751b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 626e808943c54503bc4bd8cf1cd3e4b5 in term 1.
I20240501 14:16:15.520462 16880 raft_consensus.cc:2454] T 5554988a4ab04a51abb784a41999e1a4 P e1a9c39d088a43e7b6f9beee2068deeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 626e808943c54503bc4bd8cf1cd3e4b5 in term 1.
I20240501 14:16:15.520648 16683 leader_election.cc:302] T 5554988a4ab04a51abb784a41999e1a4 P 626e808943c54503bc4bd8cf1cd3e4b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48789b208dcf4f8ab388470fa43a751b, 626e808943c54503bc4bd8cf1cd3e4b5; no voters:
I20240501 14:16:15.520892 17219 raft_consensus.cc:2781] T 5554988a4ab04a51abb784a41999e1a4 P 626e808943c54503bc4bd8cf1cd3e4b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:16:15.521075 17219 raft_consensus.cc:686] T 5554988a4ab04a51abb784a41999e1a4 P 626e808943c54503bc4bd8cf1cd3e4b5 [term 1 LEADER]: Becoming Leader. State: Replica: 626e808943c54503bc4bd8cf1cd3e4b5, State: Running, Role: LEADER
I20240501 14:16:15.521349 17219 consensus_queue.cc:237] T 5554988a4ab04a51abb784a41999e1a4 P 626e808943c54503bc4bd8cf1cd3e4b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:15.524503 16606 catalog_manager.cc:5484] T 5554988a4ab04a51abb784a41999e1a4 P 626e808943c54503bc4bd8cf1cd3e4b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 626e808943c54503bc4bd8cf1cd3e4b5 (127.7.39.129). New cstate: current_term: 1 leader_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } health_report { overall_health: UNKNOWN } } }
W20240501 14:16:15.552736 16927 tablet.cc:2369] T 5554988a4ab04a51abb784a41999e1a4 P e1a9c39d088a43e7b6f9beee2068deeb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:16:15.601482 17216 raft_consensus.cc:482] T 75227f95427f4c6d8dc2b31b5f4072e9 P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:16:15.601645 17216 raft_consensus.cc:504] T 75227f95427f4c6d8dc2b31b5f4072e9 P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:15.602922 17216 leader_election.cc:288] T 75227f95427f4c6d8dc2b31b5f4072e9 P 48789b208dcf4f8ab388470fa43a751b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1a9c39d088a43e7b6f9beee2068deeb (127.7.39.130:40479), 626e808943c54503bc4bd8cf1cd3e4b5 (127.7.39.129:43915)
I20240501 14:16:15.606670 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75227f95427f4c6d8dc2b31b5f4072e9" candidate_uuid: "48789b208dcf4f8ab388470fa43a751b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" is_pre_election: true
I20240501 14:16:15.606884 16748 raft_consensus.cc:2454] T 75227f95427f4c6d8dc2b31b5f4072e9 P 626e808943c54503bc4bd8cf1cd3e4b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48789b208dcf4f8ab388470fa43a751b in term 0.
I20240501 14:16:15.607026 16880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75227f95427f4c6d8dc2b31b5f4072e9" candidate_uuid: "48789b208dcf4f8ab388470fa43a751b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" is_pre_election: true
I20240501 14:16:15.607198 16880 raft_consensus.cc:2454] T 75227f95427f4c6d8dc2b31b5f4072e9 P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48789b208dcf4f8ab388470fa43a751b in term 0.
I20240501 14:16:15.607390 16947 leader_election.cc:302] T 75227f95427f4c6d8dc2b31b5f4072e9 P 48789b208dcf4f8ab388470fa43a751b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48789b208dcf4f8ab388470fa43a751b, 626e808943c54503bc4bd8cf1cd3e4b5; no voters:
I20240501 14:16:15.607679 17216 raft_consensus.cc:2781] T 75227f95427f4c6d8dc2b31b5f4072e9 P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:16:15.607744 17216 raft_consensus.cc:482] T 75227f95427f4c6d8dc2b31b5f4072e9 P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:16:15.607797 17216 raft_consensus.cc:3037] T 75227f95427f4c6d8dc2b31b5f4072e9 P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:15.611481 17216 raft_consensus.cc:504] T 75227f95427f4c6d8dc2b31b5f4072e9 P 48789b208dcf4f8ab388470fa43a751b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:15.612190 17216 leader_election.cc:288] T 75227f95427f4c6d8dc2b31b5f4072e9 P 48789b208dcf4f8ab388470fa43a751b [CANDIDATE]: Term 1 election: Requested vote from peers e1a9c39d088a43e7b6f9beee2068deeb (127.7.39.130:40479), 626e808943c54503bc4bd8cf1cd3e4b5 (127.7.39.129:43915)
I20240501 14:16:15.612614 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75227f95427f4c6d8dc2b31b5f4072e9" candidate_uuid: "48789b208dcf4f8ab388470fa43a751b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "626e808943c54503bc4bd8cf1cd3e4b5"
I20240501 14:16:15.612648 16880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75227f95427f4c6d8dc2b31b5f4072e9" candidate_uuid: "48789b208dcf4f8ab388470fa43a751b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9c39d088a43e7b6f9beee2068deeb"
I20240501 14:16:15.612725 16748 raft_consensus.cc:3037] T 75227f95427f4c6d8dc2b31b5f4072e9 P 626e808943c54503bc4bd8cf1cd3e4b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:15.612776 16880 raft_consensus.cc:3037] T 75227f95427f4c6d8dc2b31b5f4072e9 P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:15.613569 16748 raft_consensus.cc:2454] T 75227f95427f4c6d8dc2b31b5f4072e9 P 626e808943c54503bc4bd8cf1cd3e4b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48789b208dcf4f8ab388470fa43a751b in term 1.
I20240501 14:16:15.613567 16880 raft_consensus.cc:2454] T 75227f95427f4c6d8dc2b31b5f4072e9 P e1a9c39d088a43e7b6f9beee2068deeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48789b208dcf4f8ab388470fa43a751b in term 1.
I20240501 14:16:15.613871 16947 leader_election.cc:302] T 75227f95427f4c6d8dc2b31b5f4072e9 P 48789b208dcf4f8ab388470fa43a751b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48789b208dcf4f8ab388470fa43a751b, 626e808943c54503bc4bd8cf1cd3e4b5; no voters:
I20240501 14:16:15.614032 17216 raft_consensus.cc:2781] T 75227f95427f4c6d8dc2b31b5f4072e9 P 48789b208dcf4f8ab388470fa43a751b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:16:15.614245 17216 raft_consensus.cc:686] T 75227f95427f4c6d8dc2b31b5f4072e9 P 48789b208dcf4f8ab388470fa43a751b [term 1 LEADER]: Becoming Leader. State: Replica: 48789b208dcf4f8ab388470fa43a751b, State: Running, Role: LEADER
I20240501 14:16:15.614562 17216 consensus_queue.cc:237] T 75227f95427f4c6d8dc2b31b5f4072e9 P 48789b208dcf4f8ab388470fa43a751b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:15.617550 16606 catalog_manager.cc:5484] T 75227f95427f4c6d8dc2b31b5f4072e9 P 48789b208dcf4f8ab388470fa43a751b reported cstate change: term changed from 0 to 1, leader changed from <none> to 48789b208dcf4f8ab388470fa43a751b (127.7.39.131). New cstate: current_term: 1 leader_uuid: "48789b208dcf4f8ab388470fa43a751b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } health_report { overall_health: HEALTHY } } }
W20240501 14:16:15.631736 16795 tablet.cc:2369] T 5554988a4ab04a51abb784a41999e1a4 P 626e808943c54503bc4bd8cf1cd3e4b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:16:15.712496 17059 tablet.cc:2369] T 5554988a4ab04a51abb784a41999e1a4 P 48789b208dcf4f8ab388470fa43a751b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:16:15.786370 17216 raft_consensus.cc:482] T cbcda590e51c480ba98ddaf9370c34fa P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:16:15.786521 17216 raft_consensus.cc:504] T cbcda590e51c480ba98ddaf9370c34fa P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:15.786968 17216 leader_election.cc:288] T cbcda590e51c480ba98ddaf9370c34fa P 48789b208dcf4f8ab388470fa43a751b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 626e808943c54503bc4bd8cf1cd3e4b5 (127.7.39.129:43915), e1a9c39d088a43e7b6f9beee2068deeb (127.7.39.130:40479)
I20240501 14:16:15.787510 16880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbcda590e51c480ba98ddaf9370c34fa" candidate_uuid: "48789b208dcf4f8ab388470fa43a751b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" is_pre_election: true
I20240501 14:16:15.787506 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbcda590e51c480ba98ddaf9370c34fa" candidate_uuid: "48789b208dcf4f8ab388470fa43a751b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" is_pre_election: true
I20240501 14:16:15.787650 16880 raft_consensus.cc:2454] T cbcda590e51c480ba98ddaf9370c34fa P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48789b208dcf4f8ab388470fa43a751b in term 0.
I20240501 14:16:15.787671 16748 raft_consensus.cc:2454] T cbcda590e51c480ba98ddaf9370c34fa P 626e808943c54503bc4bd8cf1cd3e4b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48789b208dcf4f8ab388470fa43a751b in term 0.
I20240501 14:16:15.788002 16948 leader_election.cc:302] T cbcda590e51c480ba98ddaf9370c34fa P 48789b208dcf4f8ab388470fa43a751b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48789b208dcf4f8ab388470fa43a751b, e1a9c39d088a43e7b6f9beee2068deeb; no voters:
I20240501 14:16:15.788178 17216 raft_consensus.cc:2781] T cbcda590e51c480ba98ddaf9370c34fa P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:16:15.788252 17216 raft_consensus.cc:482] T cbcda590e51c480ba98ddaf9370c34fa P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:16:15.788292 17216 raft_consensus.cc:3037] T cbcda590e51c480ba98ddaf9370c34fa P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:15.789109 17216 raft_consensus.cc:504] T cbcda590e51c480ba98ddaf9370c34fa P 48789b208dcf4f8ab388470fa43a751b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:15.789427 17216 leader_election.cc:288] T cbcda590e51c480ba98ddaf9370c34fa P 48789b208dcf4f8ab388470fa43a751b [CANDIDATE]: Term 1 election: Requested vote from peers 626e808943c54503bc4bd8cf1cd3e4b5 (127.7.39.129:43915), e1a9c39d088a43e7b6f9beee2068deeb (127.7.39.130:40479)
I20240501 14:16:15.789770 16880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbcda590e51c480ba98ddaf9370c34fa" candidate_uuid: "48789b208dcf4f8ab388470fa43a751b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9c39d088a43e7b6f9beee2068deeb"
I20240501 14:16:15.789781 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbcda590e51c480ba98ddaf9370c34fa" candidate_uuid: "48789b208dcf4f8ab388470fa43a751b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "626e808943c54503bc4bd8cf1cd3e4b5"
I20240501 14:16:15.789882 16880 raft_consensus.cc:3037] T cbcda590e51c480ba98ddaf9370c34fa P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:15.789901 16748 raft_consensus.cc:3037] T cbcda590e51c480ba98ddaf9370c34fa P 626e808943c54503bc4bd8cf1cd3e4b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:15.790617 16748 raft_consensus.cc:2454] T cbcda590e51c480ba98ddaf9370c34fa P 626e808943c54503bc4bd8cf1cd3e4b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48789b208dcf4f8ab388470fa43a751b in term 1.
I20240501 14:16:15.790627 16880 raft_consensus.cc:2454] T cbcda590e51c480ba98ddaf9370c34fa P e1a9c39d088a43e7b6f9beee2068deeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48789b208dcf4f8ab388470fa43a751b in term 1.
I20240501 14:16:15.790902 16948 leader_election.cc:302] T cbcda590e51c480ba98ddaf9370c34fa P 48789b208dcf4f8ab388470fa43a751b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48789b208dcf4f8ab388470fa43a751b, e1a9c39d088a43e7b6f9beee2068deeb; no voters:
I20240501 14:16:15.791085 17216 raft_consensus.cc:2781] T cbcda590e51c480ba98ddaf9370c34fa P 48789b208dcf4f8ab388470fa43a751b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:16:15.791178 17216 raft_consensus.cc:686] T cbcda590e51c480ba98ddaf9370c34fa P 48789b208dcf4f8ab388470fa43a751b [term 1 LEADER]: Becoming Leader. State: Replica: 48789b208dcf4f8ab388470fa43a751b, State: Running, Role: LEADER
I20240501 14:16:15.791318 17216 consensus_queue.cc:237] T cbcda590e51c480ba98ddaf9370c34fa P 48789b208dcf4f8ab388470fa43a751b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:15.792883 16606 catalog_manager.cc:5484] T cbcda590e51c480ba98ddaf9370c34fa P 48789b208dcf4f8ab388470fa43a751b reported cstate change: term changed from 0 to 1, leader changed from <none> to 48789b208dcf4f8ab388470fa43a751b (127.7.39.131). New cstate: current_term: 1 leader_uuid: "48789b208dcf4f8ab388470fa43a751b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } health_report { overall_health: HEALTHY } } }
W20240501 14:16:15.923519 17233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:16:15.923774 17233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:16:15.926796 17233 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240501 14:16:15.964735 17219 consensus_queue.cc:1035] T 5554988a4ab04a51abb784a41999e1a4 P 626e808943c54503bc4bd8cf1cd3e4b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:16:15.975004 17226 consensus_queue.cc:1035] T 5554988a4ab04a51abb784a41999e1a4 P 626e808943c54503bc4bd8cf1cd3e4b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:16:16.016031 17216 consensus_queue.cc:1035] T 75227f95427f4c6d8dc2b31b5f4072e9 P 48789b208dcf4f8ab388470fa43a751b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:16:16.021576 17230 consensus_queue.cc:1035] T 75227f95427f4c6d8dc2b31b5f4072e9 P 48789b208dcf4f8ab388470fa43a751b [LEADER]: Connected to new peer: Peer: permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:16:16.023430 16747 raft_consensus.cc:1261] T cbcda590e51c480ba98ddaf9370c34fa P 626e808943c54503bc4bd8cf1cd3e4b5 [term 1 FOLLOWER]: Refusing update from remote peer 48789b208dcf4f8ab388470fa43a751b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:16:16.023836 16879 raft_consensus.cc:1261] T cbcda590e51c480ba98ddaf9370c34fa P e1a9c39d088a43e7b6f9beee2068deeb [term 1 FOLLOWER]: Refusing update from remote peer 48789b208dcf4f8ab388470fa43a751b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:16:16.025763 17216 consensus_queue.cc:1035] T cbcda590e51c480ba98ddaf9370c34fa P 48789b208dcf4f8ab388470fa43a751b [LEADER]: Connected to new peer: Peer: permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:16:16.026348 17216 consensus_queue.cc:1035] T cbcda590e51c480ba98ddaf9370c34fa P 48789b208dcf4f8ab388470fa43a751b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:16:16.036406 17271 mvcc.cc:205] Tried to move back new op lower bound from 7022890909782900736 to 7022890908119982080. Current Snapshot: MvccSnapshot[applied={T|T < 7022890909782900736}]
I20240501 14:16:16.051172 17280 mvcc.cc:205] Tried to move back new op lower bound from 7022890909772771328 to 7022890908842373120. Current Snapshot: MvccSnapshot[applied={T|T < 7022890909772771328}]
INSERT report
rows total: 2000
time total: 130.324 ms
time per row: 0.0651618 ms
I20240501 14:16:16.099865 16605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36462:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240501 14:16:16.100420 16605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 14:16:16.105800 16859 tablet_service.cc:1467] Processing CreateTablet for tablet bd10315524d7434a88b3bb4b2a227952 (DEFAULT_TABLE table=rebalance_test_table_1 [id=cd8feb7a0a1849e3b796c99b851252fc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:16.105866 16726 tablet_service.cc:1467] Processing CreateTablet for tablet bd10315524d7434a88b3bb4b2a227952 (DEFAULT_TABLE table=rebalance_test_table_1 [id=cd8feb7a0a1849e3b796c99b851252fc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:16.106184 16859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd10315524d7434a88b3bb4b2a227952. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:16.106204 16726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd10315524d7434a88b3bb4b2a227952. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:16.106241 16860 tablet_service.cc:1467] Processing CreateTablet for tablet e82af677f1f644549a3213b4b8bf3d4f (DEFAULT_TABLE table=rebalance_test_table_1 [id=cd8feb7a0a1849e3b796c99b851252fc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:16.106262 16858 tablet_service.cc:1467] Processing CreateTablet for tablet 5639d52050b8423393104b34c9d970ac (DEFAULT_TABLE table=rebalance_test_table_1 [id=cd8feb7a0a1849e3b796c99b851252fc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:16.106438 16860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e82af677f1f644549a3213b4b8bf3d4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:16.106647 16858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5639d52050b8423393104b34c9d970ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:16.108651 17286 tablet_bootstrap.cc:492] T bd10315524d7434a88b3bb4b2a227952 P 626e808943c54503bc4bd8cf1cd3e4b5: Bootstrap starting.
I20240501 14:16:16.108897 16992 tablet_service.cc:1467] Processing CreateTablet for tablet bd10315524d7434a88b3bb4b2a227952 (DEFAULT_TABLE table=rebalance_test_table_1 [id=cd8feb7a0a1849e3b796c99b851252fc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:16.109207 16992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd10315524d7434a88b3bb4b2a227952. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:16.109695 17286 tablet_bootstrap.cc:654] T bd10315524d7434a88b3bb4b2a227952 P 626e808943c54503bc4bd8cf1cd3e4b5: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:16.109802 16728 tablet_service.cc:1467] Processing CreateTablet for tablet e82af677f1f644549a3213b4b8bf3d4f (DEFAULT_TABLE table=rebalance_test_table_1 [id=cd8feb7a0a1849e3b796c99b851252fc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:16.110047 16728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e82af677f1f644549a3213b4b8bf3d4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:16.110208 17285 tablet_bootstrap.cc:492] T e82af677f1f644549a3213b4b8bf3d4f P e1a9c39d088a43e7b6f9beee2068deeb: Bootstrap starting.
I20240501 14:16:16.111204 17285 tablet_bootstrap.cc:654] T e82af677f1f644549a3213b4b8bf3d4f P e1a9c39d088a43e7b6f9beee2068deeb: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:16.111567 17288 tablet_bootstrap.cc:492] T bd10315524d7434a88b3bb4b2a227952 P 48789b208dcf4f8ab388470fa43a751b: Bootstrap starting.
I20240501 14:16:16.112399 17288 tablet_bootstrap.cc:654] T bd10315524d7434a88b3bb4b2a227952 P 48789b208dcf4f8ab388470fa43a751b: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:16.112686 16992 tablet_service.cc:1467] Processing CreateTablet for tablet 5639d52050b8423393104b34c9d970ac (DEFAULT_TABLE table=rebalance_test_table_1 [id=cd8feb7a0a1849e3b796c99b851252fc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:16.112905 17285 tablet_bootstrap.cc:492] T e82af677f1f644549a3213b4b8bf3d4f P e1a9c39d088a43e7b6f9beee2068deeb: No bootstrap required, opened a new log
I20240501 14:16:16.112926 16992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5639d52050b8423393104b34c9d970ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:16.113018 17285 ts_tablet_manager.cc:1367] T e82af677f1f644549a3213b4b8bf3d4f P e1a9c39d088a43e7b6f9beee2068deeb: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:16:16.113480 17285 raft_consensus.cc:348] T e82af677f1f644549a3213b4b8bf3d4f P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.113605 17285 raft_consensus.cc:374] T e82af677f1f644549a3213b4b8bf3d4f P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:16.113649 17285 raft_consensus.cc:729] T e82af677f1f644549a3213b4b8bf3d4f P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a9c39d088a43e7b6f9beee2068deeb, State: Initialized, Role: FOLLOWER
I20240501 14:16:16.113796 17285 consensus_queue.cc:260] T e82af677f1f644549a3213b4b8bf3d4f P e1a9c39d088a43e7b6f9beee2068deeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.114105 17285 ts_tablet_manager.cc:1398] T e82af677f1f644549a3213b4b8bf3d4f P e1a9c39d088a43e7b6f9beee2068deeb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:16:16.112375 16990 tablet_service.cc:1467] Processing CreateTablet for tablet e82af677f1f644549a3213b4b8bf3d4f (DEFAULT_TABLE table=rebalance_test_table_1 [id=cd8feb7a0a1849e3b796c99b851252fc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:16.114327 17285 tablet_bootstrap.cc:492] T bd10315524d7434a88b3bb4b2a227952 P e1a9c39d088a43e7b6f9beee2068deeb: Bootstrap starting.
I20240501 14:16:16.114393 16990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e82af677f1f644549a3213b4b8bf3d4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:16.115263 17285 tablet_bootstrap.cc:654] T bd10315524d7434a88b3bb4b2a227952 P e1a9c39d088a43e7b6f9beee2068deeb: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:16.116276 16726 tablet_service.cc:1467] Processing CreateTablet for tablet 5639d52050b8423393104b34c9d970ac (DEFAULT_TABLE table=rebalance_test_table_1 [id=cd8feb7a0a1849e3b796c99b851252fc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:16.116500 16726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5639d52050b8423393104b34c9d970ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:16.116576 17285 tablet_bootstrap.cc:492] T bd10315524d7434a88b3bb4b2a227952 P e1a9c39d088a43e7b6f9beee2068deeb: No bootstrap required, opened a new log
I20240501 14:16:16.116653 17285 ts_tablet_manager.cc:1367] T bd10315524d7434a88b3bb4b2a227952 P e1a9c39d088a43e7b6f9beee2068deeb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:16:16.117092 17285 raft_consensus.cc:348] T bd10315524d7434a88b3bb4b2a227952 P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.117204 17285 raft_consensus.cc:374] T bd10315524d7434a88b3bb4b2a227952 P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:16.117234 17285 raft_consensus.cc:729] T bd10315524d7434a88b3bb4b2a227952 P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a9c39d088a43e7b6f9beee2068deeb, State: Initialized, Role: FOLLOWER
I20240501 14:16:16.117383 17285 consensus_queue.cc:260] T bd10315524d7434a88b3bb4b2a227952 P e1a9c39d088a43e7b6f9beee2068deeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.117651 17285 ts_tablet_manager.cc:1398] T bd10315524d7434a88b3bb4b2a227952 P e1a9c39d088a43e7b6f9beee2068deeb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:16:16.117787 17285 tablet_bootstrap.cc:492] T 5639d52050b8423393104b34c9d970ac P e1a9c39d088a43e7b6f9beee2068deeb: Bootstrap starting.
I20240501 14:16:16.118631 17285 tablet_bootstrap.cc:654] T 5639d52050b8423393104b34c9d970ac P e1a9c39d088a43e7b6f9beee2068deeb: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:16.119025 17288 tablet_bootstrap.cc:492] T bd10315524d7434a88b3bb4b2a227952 P 48789b208dcf4f8ab388470fa43a751b: No bootstrap required, opened a new log
I20240501 14:16:16.119072 17286 tablet_bootstrap.cc:492] T bd10315524d7434a88b3bb4b2a227952 P 626e808943c54503bc4bd8cf1cd3e4b5: No bootstrap required, opened a new log
I20240501 14:16:16.119160 17286 ts_tablet_manager.cc:1367] T bd10315524d7434a88b3bb4b2a227952 P 626e808943c54503bc4bd8cf1cd3e4b5: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 14:16:16.119171 17288 ts_tablet_manager.cc:1367] T bd10315524d7434a88b3bb4b2a227952 P 48789b208dcf4f8ab388470fa43a751b: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240501 14:16:16.119647 17288 raft_consensus.cc:348] T bd10315524d7434a88b3bb4b2a227952 P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.119840 17288 raft_consensus.cc:374] T bd10315524d7434a88b3bb4b2a227952 P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:16.119895 17288 raft_consensus.cc:729] T bd10315524d7434a88b3bb4b2a227952 P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48789b208dcf4f8ab388470fa43a751b, State: Initialized, Role: FOLLOWER
I20240501 14:16:16.120043 17288 consensus_queue.cc:260] T bd10315524d7434a88b3bb4b2a227952 P 48789b208dcf4f8ab388470fa43a751b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.120165 17285 tablet_bootstrap.cc:492] T 5639d52050b8423393104b34c9d970ac P e1a9c39d088a43e7b6f9beee2068deeb: No bootstrap required, opened a new log
I20240501 14:16:16.120121 17286 raft_consensus.cc:348] T bd10315524d7434a88b3bb4b2a227952 P 626e808943c54503bc4bd8cf1cd3e4b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.120230 17286 raft_consensus.cc:374] T bd10315524d7434a88b3bb4b2a227952 P 626e808943c54503bc4bd8cf1cd3e4b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:16.120260 17286 raft_consensus.cc:729] T bd10315524d7434a88b3bb4b2a227952 P 626e808943c54503bc4bd8cf1cd3e4b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 626e808943c54503bc4bd8cf1cd3e4b5, State: Initialized, Role: FOLLOWER
I20240501 14:16:16.120358 17288 ts_tablet_manager.cc:1398] T bd10315524d7434a88b3bb4b2a227952 P 48789b208dcf4f8ab388470fa43a751b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:16:16.120385 17286 consensus_queue.cc:260] T bd10315524d7434a88b3bb4b2a227952 P 626e808943c54503bc4bd8cf1cd3e4b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.120504 17285 ts_tablet_manager.cc:1367] T 5639d52050b8423393104b34c9d970ac P e1a9c39d088a43e7b6f9beee2068deeb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:16:16.120537 17288 tablet_bootstrap.cc:492] T 5639d52050b8423393104b34c9d970ac P 48789b208dcf4f8ab388470fa43a751b: Bootstrap starting.
I20240501 14:16:16.120760 17286 ts_tablet_manager.cc:1398] T bd10315524d7434a88b3bb4b2a227952 P 626e808943c54503bc4bd8cf1cd3e4b5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:16:16.120949 17286 tablet_bootstrap.cc:492] T e82af677f1f644549a3213b4b8bf3d4f P 626e808943c54503bc4bd8cf1cd3e4b5: Bootstrap starting.
I20240501 14:16:16.120978 17285 raft_consensus.cc:348] T 5639d52050b8423393104b34c9d970ac P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.121102 17285 raft_consensus.cc:374] T 5639d52050b8423393104b34c9d970ac P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:16.121151 17285 raft_consensus.cc:729] T 5639d52050b8423393104b34c9d970ac P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a9c39d088a43e7b6f9beee2068deeb, State: Initialized, Role: FOLLOWER
I20240501 14:16:16.121294 17285 consensus_queue.cc:260] T 5639d52050b8423393104b34c9d970ac P e1a9c39d088a43e7b6f9beee2068deeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.121616 17285 ts_tablet_manager.cc:1398] T 5639d52050b8423393104b34c9d970ac P e1a9c39d088a43e7b6f9beee2068deeb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:16:16.122262 17288 tablet_bootstrap.cc:654] T 5639d52050b8423393104b34c9d970ac P 48789b208dcf4f8ab388470fa43a751b: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:16.123943 17288 tablet_bootstrap.cc:492] T 5639d52050b8423393104b34c9d970ac P 48789b208dcf4f8ab388470fa43a751b: No bootstrap required, opened a new log
I20240501 14:16:16.124073 17288 ts_tablet_manager.cc:1367] T 5639d52050b8423393104b34c9d970ac P 48789b208dcf4f8ab388470fa43a751b: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:16:16.124148 17286 tablet_bootstrap.cc:654] T e82af677f1f644549a3213b4b8bf3d4f P 626e808943c54503bc4bd8cf1cd3e4b5: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:16.124509 17288 raft_consensus.cc:348] T 5639d52050b8423393104b34c9d970ac P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.124697 17288 raft_consensus.cc:374] T 5639d52050b8423393104b34c9d970ac P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:16.124783 17288 raft_consensus.cc:729] T 5639d52050b8423393104b34c9d970ac P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48789b208dcf4f8ab388470fa43a751b, State: Initialized, Role: FOLLOWER
I20240501 14:16:16.124926 17288 consensus_queue.cc:260] T 5639d52050b8423393104b34c9d970ac P 48789b208dcf4f8ab388470fa43a751b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.125293 17286 tablet_bootstrap.cc:492] T e82af677f1f644549a3213b4b8bf3d4f P 626e808943c54503bc4bd8cf1cd3e4b5: No bootstrap required, opened a new log
I20240501 14:16:16.125315 17288 ts_tablet_manager.cc:1398] T 5639d52050b8423393104b34c9d970ac P 48789b208dcf4f8ab388470fa43a751b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:16:16.125386 17286 ts_tablet_manager.cc:1367] T e82af677f1f644549a3213b4b8bf3d4f P 626e808943c54503bc4bd8cf1cd3e4b5: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:16:16.125494 17288 tablet_bootstrap.cc:492] T e82af677f1f644549a3213b4b8bf3d4f P 48789b208dcf4f8ab388470fa43a751b: Bootstrap starting.
I20240501 14:16:16.125856 17286 raft_consensus.cc:348] T e82af677f1f644549a3213b4b8bf3d4f P 626e808943c54503bc4bd8cf1cd3e4b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.125972 17286 raft_consensus.cc:374] T e82af677f1f644549a3213b4b8bf3d4f P 626e808943c54503bc4bd8cf1cd3e4b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:16.126017 17286 raft_consensus.cc:729] T e82af677f1f644549a3213b4b8bf3d4f P 626e808943c54503bc4bd8cf1cd3e4b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 626e808943c54503bc4bd8cf1cd3e4b5, State: Initialized, Role: FOLLOWER
I20240501 14:16:16.126139 17286 consensus_queue.cc:260] T e82af677f1f644549a3213b4b8bf3d4f P 626e808943c54503bc4bd8cf1cd3e4b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.126473 17288 tablet_bootstrap.cc:654] T e82af677f1f644549a3213b4b8bf3d4f P 48789b208dcf4f8ab388470fa43a751b: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:16.126473 17286 ts_tablet_manager.cc:1398] T e82af677f1f644549a3213b4b8bf3d4f P 626e808943c54503bc4bd8cf1cd3e4b5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:16:16.126660 17286 tablet_bootstrap.cc:492] T 5639d52050b8423393104b34c9d970ac P 626e808943c54503bc4bd8cf1cd3e4b5: Bootstrap starting.
I20240501 14:16:16.128088 17286 tablet_bootstrap.cc:654] T 5639d52050b8423393104b34c9d970ac P 626e808943c54503bc4bd8cf1cd3e4b5: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:16.129195 17288 tablet_bootstrap.cc:492] T e82af677f1f644549a3213b4b8bf3d4f P 48789b208dcf4f8ab388470fa43a751b: No bootstrap required, opened a new log
I20240501 14:16:16.129303 17288 ts_tablet_manager.cc:1367] T e82af677f1f644549a3213b4b8bf3d4f P 48789b208dcf4f8ab388470fa43a751b: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:16:16.129825 17286 tablet_bootstrap.cc:492] T 5639d52050b8423393104b34c9d970ac P 626e808943c54503bc4bd8cf1cd3e4b5: No bootstrap required, opened a new log
I20240501 14:16:16.129778 17288 raft_consensus.cc:348] T e82af677f1f644549a3213b4b8bf3d4f P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.129879 17288 raft_consensus.cc:374] T e82af677f1f644549a3213b4b8bf3d4f P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:16.129915 17286 ts_tablet_manager.cc:1367] T 5639d52050b8423393104b34c9d970ac P 626e808943c54503bc4bd8cf1cd3e4b5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:16:16.129920 17288 raft_consensus.cc:729] T e82af677f1f644549a3213b4b8bf3d4f P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48789b208dcf4f8ab388470fa43a751b, State: Initialized, Role: FOLLOWER
I20240501 14:16:16.130055 17288 consensus_queue.cc:260] T e82af677f1f644549a3213b4b8bf3d4f P 48789b208dcf4f8ab388470fa43a751b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.130321 17288 ts_tablet_manager.cc:1398] T e82af677f1f644549a3213b4b8bf3d4f P 48789b208dcf4f8ab388470fa43a751b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:16:16.130326 17286 raft_consensus.cc:348] T 5639d52050b8423393104b34c9d970ac P 626e808943c54503bc4bd8cf1cd3e4b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.130425 17286 raft_consensus.cc:374] T 5639d52050b8423393104b34c9d970ac P 626e808943c54503bc4bd8cf1cd3e4b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:16.130466 17286 raft_consensus.cc:729] T 5639d52050b8423393104b34c9d970ac P 626e808943c54503bc4bd8cf1cd3e4b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 626e808943c54503bc4bd8cf1cd3e4b5, State: Initialized, Role: FOLLOWER
I20240501 14:16:16.130584 17286 consensus_queue.cc:260] T 5639d52050b8423393104b34c9d970ac P 626e808943c54503bc4bd8cf1cd3e4b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.130904 17286 ts_tablet_manager.cc:1398] T 5639d52050b8423393104b34c9d970ac P 626e808943c54503bc4bd8cf1cd3e4b5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:16:16.190622 17268 raft_consensus.cc:482] T e82af677f1f644549a3213b4b8bf3d4f P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:16:16.190778 17268 raft_consensus.cc:504] T e82af677f1f644549a3213b4b8bf3d4f P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.191255 17268 leader_election.cc:288] T e82af677f1f644549a3213b4b8bf3d4f P 48789b208dcf4f8ab388470fa43a751b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1a9c39d088a43e7b6f9beee2068deeb (127.7.39.130:40479), 626e808943c54503bc4bd8cf1cd3e4b5 (127.7.39.129:43915)
I20240501 14:16:16.191733 16880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e82af677f1f644549a3213b4b8bf3d4f" candidate_uuid: "48789b208dcf4f8ab388470fa43a751b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" is_pre_election: true
I20240501 14:16:16.191879 16880 raft_consensus.cc:2454] T e82af677f1f644549a3213b4b8bf3d4f P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48789b208dcf4f8ab388470fa43a751b in term 0.
I20240501 14:16:16.191998 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e82af677f1f644549a3213b4b8bf3d4f" candidate_uuid: "48789b208dcf4f8ab388470fa43a751b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" is_pre_election: true
I20240501 14:16:16.192191 16748 raft_consensus.cc:2454] T e82af677f1f644549a3213b4b8bf3d4f P 626e808943c54503bc4bd8cf1cd3e4b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48789b208dcf4f8ab388470fa43a751b in term 0.
I20240501 14:16:16.192234 16948 leader_election.cc:302] T e82af677f1f644549a3213b4b8bf3d4f P 48789b208dcf4f8ab388470fa43a751b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48789b208dcf4f8ab388470fa43a751b, e1a9c39d088a43e7b6f9beee2068deeb; no voters:
I20240501 14:16:16.192414 17268 raft_consensus.cc:2781] T e82af677f1f644549a3213b4b8bf3d4f P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:16:16.192479 17268 raft_consensus.cc:482] T e82af677f1f644549a3213b4b8bf3d4f P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:16:16.192517 17268 raft_consensus.cc:3037] T e82af677f1f644549a3213b4b8bf3d4f P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:16.193425 17268 raft_consensus.cc:504] T e82af677f1f644549a3213b4b8bf3d4f P 48789b208dcf4f8ab388470fa43a751b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.193720 17268 leader_election.cc:288] T e82af677f1f644549a3213b4b8bf3d4f P 48789b208dcf4f8ab388470fa43a751b [CANDIDATE]: Term 1 election: Requested vote from peers e1a9c39d088a43e7b6f9beee2068deeb (127.7.39.130:40479), 626e808943c54503bc4bd8cf1cd3e4b5 (127.7.39.129:43915)
I20240501 14:16:16.194031 16880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e82af677f1f644549a3213b4b8bf3d4f" candidate_uuid: "48789b208dcf4f8ab388470fa43a751b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9c39d088a43e7b6f9beee2068deeb"
I20240501 14:16:16.194056 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e82af677f1f644549a3213b4b8bf3d4f" candidate_uuid: "48789b208dcf4f8ab388470fa43a751b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "626e808943c54503bc4bd8cf1cd3e4b5"
I20240501 14:16:16.194141 16748 raft_consensus.cc:3037] T e82af677f1f644549a3213b4b8bf3d4f P 626e808943c54503bc4bd8cf1cd3e4b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:16.194136 16880 raft_consensus.cc:3037] T e82af677f1f644549a3213b4b8bf3d4f P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:16.194902 16748 raft_consensus.cc:2454] T e82af677f1f644549a3213b4b8bf3d4f P 626e808943c54503bc4bd8cf1cd3e4b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48789b208dcf4f8ab388470fa43a751b in term 1.
I20240501 14:16:16.194967 16880 raft_consensus.cc:2454] T e82af677f1f644549a3213b4b8bf3d4f P e1a9c39d088a43e7b6f9beee2068deeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48789b208dcf4f8ab388470fa43a751b in term 1.
I20240501 14:16:16.195185 16947 leader_election.cc:302] T e82af677f1f644549a3213b4b8bf3d4f P 48789b208dcf4f8ab388470fa43a751b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48789b208dcf4f8ab388470fa43a751b, 626e808943c54503bc4bd8cf1cd3e4b5; no voters:
I20240501 14:16:16.195376 17268 raft_consensus.cc:2781] T e82af677f1f644549a3213b4b8bf3d4f P 48789b208dcf4f8ab388470fa43a751b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:16:16.195494 17268 raft_consensus.cc:686] T e82af677f1f644549a3213b4b8bf3d4f P 48789b208dcf4f8ab388470fa43a751b [term 1 LEADER]: Becoming Leader. State: Replica: 48789b208dcf4f8ab388470fa43a751b, State: Running, Role: LEADER
I20240501 14:16:16.195642 17268 consensus_queue.cc:237] T e82af677f1f644549a3213b4b8bf3d4f P 48789b208dcf4f8ab388470fa43a751b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.197412 16606 catalog_manager.cc:5484] T e82af677f1f644549a3213b4b8bf3d4f P 48789b208dcf4f8ab388470fa43a751b reported cstate change: term changed from 0 to 1, leader changed from <none> to 48789b208dcf4f8ab388470fa43a751b (127.7.39.131). New cstate: current_term: 1 leader_uuid: "48789b208dcf4f8ab388470fa43a751b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } health_report { overall_health: HEALTHY } } }
I20240501 14:16:16.435464 17215 raft_consensus.cc:482] T 5639d52050b8423393104b34c9d970ac P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:16:16.435613 17215 raft_consensus.cc:504] T 5639d52050b8423393104b34c9d970ac P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.436672 17215 leader_election.cc:288] T 5639d52050b8423393104b34c9d970ac P e1a9c39d088a43e7b6f9beee2068deeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 626e808943c54503bc4bd8cf1cd3e4b5 (127.7.39.129:43915), 48789b208dcf4f8ab388470fa43a751b (127.7.39.131:45011)
I20240501 14:16:16.441471 17012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5639d52050b8423393104b34c9d970ac" candidate_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48789b208dcf4f8ab388470fa43a751b" is_pre_election: true
I20240501 14:16:16.441629 17012 raft_consensus.cc:2454] T 5639d52050b8423393104b34c9d970ac P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1a9c39d088a43e7b6f9beee2068deeb in term 0.
I20240501 14:16:16.441831 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5639d52050b8423393104b34c9d970ac" candidate_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" is_pre_election: true
I20240501 14:16:16.441951 16748 raft_consensus.cc:2454] T 5639d52050b8423393104b34c9d970ac P 626e808943c54503bc4bd8cf1cd3e4b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1a9c39d088a43e7b6f9beee2068deeb in term 0.
I20240501 14:16:16.441972 16815 leader_election.cc:302] T 5639d52050b8423393104b34c9d970ac P e1a9c39d088a43e7b6f9beee2068deeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48789b208dcf4f8ab388470fa43a751b, e1a9c39d088a43e7b6f9beee2068deeb; no voters:
I20240501 14:16:16.442266 17215 raft_consensus.cc:2781] T 5639d52050b8423393104b34c9d970ac P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:16:16.442320 17215 raft_consensus.cc:482] T 5639d52050b8423393104b34c9d970ac P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:16:16.442344 17215 raft_consensus.cc:3037] T 5639d52050b8423393104b34c9d970ac P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:16.443235 17215 raft_consensus.cc:504] T 5639d52050b8423393104b34c9d970ac P e1a9c39d088a43e7b6f9beee2068deeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.443583 17215 leader_election.cc:288] T 5639d52050b8423393104b34c9d970ac P e1a9c39d088a43e7b6f9beee2068deeb [CANDIDATE]: Term 1 election: Requested vote from peers 626e808943c54503bc4bd8cf1cd3e4b5 (127.7.39.129:43915), 48789b208dcf4f8ab388470fa43a751b (127.7.39.131:45011)
I20240501 14:16:16.443918 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5639d52050b8423393104b34c9d970ac" candidate_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "626e808943c54503bc4bd8cf1cd3e4b5"
I20240501 14:16:16.443954 17012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5639d52050b8423393104b34c9d970ac" candidate_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48789b208dcf4f8ab388470fa43a751b"
I20240501 14:16:16.444041 16748 raft_consensus.cc:3037] T 5639d52050b8423393104b34c9d970ac P 626e808943c54503bc4bd8cf1cd3e4b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:16.444046 17012 raft_consensus.cc:3037] T 5639d52050b8423393104b34c9d970ac P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:16.444811 17012 raft_consensus.cc:2454] T 5639d52050b8423393104b34c9d970ac P 48789b208dcf4f8ab388470fa43a751b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1a9c39d088a43e7b6f9beee2068deeb in term 1.
I20240501 14:16:16.444824 16748 raft_consensus.cc:2454] T 5639d52050b8423393104b34c9d970ac P 626e808943c54503bc4bd8cf1cd3e4b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1a9c39d088a43e7b6f9beee2068deeb in term 1.
I20240501 14:16:16.445065 16815 leader_election.cc:302] T 5639d52050b8423393104b34c9d970ac P e1a9c39d088a43e7b6f9beee2068deeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48789b208dcf4f8ab388470fa43a751b, e1a9c39d088a43e7b6f9beee2068deeb; no voters:
I20240501 14:16:16.445241 17215 raft_consensus.cc:2781] T 5639d52050b8423393104b34c9d970ac P e1a9c39d088a43e7b6f9beee2068deeb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:16:16.445444 17215 raft_consensus.cc:686] T 5639d52050b8423393104b34c9d970ac P e1a9c39d088a43e7b6f9beee2068deeb [term 1 LEADER]: Becoming Leader. State: Replica: e1a9c39d088a43e7b6f9beee2068deeb, State: Running, Role: LEADER
I20240501 14:16:16.445778 17215 consensus_queue.cc:237] T 5639d52050b8423393104b34c9d970ac P e1a9c39d088a43e7b6f9beee2068deeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.447985 16606 catalog_manager.cc:5484] T 5639d52050b8423393104b34c9d970ac P e1a9c39d088a43e7b6f9beee2068deeb reported cstate change: term changed from 0 to 1, leader changed from <none> to e1a9c39d088a43e7b6f9beee2068deeb (127.7.39.130). New cstate: current_term: 1 leader_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } health_report { overall_health: UNKNOWN } } }
I20240501 14:16:16.522797 17219 raft_consensus.cc:482] T bd10315524d7434a88b3bb4b2a227952 P 626e808943c54503bc4bd8cf1cd3e4b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:16:16.522958 17219 raft_consensus.cc:504] T bd10315524d7434a88b3bb4b2a227952 P 626e808943c54503bc4bd8cf1cd3e4b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.523481 17219 leader_election.cc:288] T bd10315524d7434a88b3bb4b2a227952 P 626e808943c54503bc4bd8cf1cd3e4b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1a9c39d088a43e7b6f9beee2068deeb (127.7.39.130:40479), 48789b208dcf4f8ab388470fa43a751b (127.7.39.131:45011)
I20240501 14:16:16.523960 16880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd10315524d7434a88b3bb4b2a227952" candidate_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" is_pre_election: true
I20240501 14:16:16.523960 17012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd10315524d7434a88b3bb4b2a227952" candidate_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48789b208dcf4f8ab388470fa43a751b" is_pre_election: true
I20240501 14:16:16.524156 16880 raft_consensus.cc:2454] T bd10315524d7434a88b3bb4b2a227952 P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 626e808943c54503bc4bd8cf1cd3e4b5 in term 0.
I20240501 14:16:16.524163 17012 raft_consensus.cc:2454] T bd10315524d7434a88b3bb4b2a227952 P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 626e808943c54503bc4bd8cf1cd3e4b5 in term 0.
I20240501 14:16:16.524502 16683 leader_election.cc:302] T bd10315524d7434a88b3bb4b2a227952 P 626e808943c54503bc4bd8cf1cd3e4b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48789b208dcf4f8ab388470fa43a751b, 626e808943c54503bc4bd8cf1cd3e4b5; no voters:
I20240501 14:16:16.524660 17219 raft_consensus.cc:2781] T bd10315524d7434a88b3bb4b2a227952 P 626e808943c54503bc4bd8cf1cd3e4b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:16:16.524716 17219 raft_consensus.cc:482] T bd10315524d7434a88b3bb4b2a227952 P 626e808943c54503bc4bd8cf1cd3e4b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:16:16.524742 17219 raft_consensus.cc:3037] T bd10315524d7434a88b3bb4b2a227952 P 626e808943c54503bc4bd8cf1cd3e4b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:16.525672 17219 raft_consensus.cc:504] T bd10315524d7434a88b3bb4b2a227952 P 626e808943c54503bc4bd8cf1cd3e4b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.526027 17219 leader_election.cc:288] T bd10315524d7434a88b3bb4b2a227952 P 626e808943c54503bc4bd8cf1cd3e4b5 [CANDIDATE]: Term 1 election: Requested vote from peers e1a9c39d088a43e7b6f9beee2068deeb (127.7.39.130:40479), 48789b208dcf4f8ab388470fa43a751b (127.7.39.131:45011)
I20240501 14:16:16.526428 16880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd10315524d7434a88b3bb4b2a227952" candidate_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9c39d088a43e7b6f9beee2068deeb"
I20240501 14:16:16.526502 17012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd10315524d7434a88b3bb4b2a227952" candidate_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48789b208dcf4f8ab388470fa43a751b"
I20240501 14:16:16.526551 16880 raft_consensus.cc:3037] T bd10315524d7434a88b3bb4b2a227952 P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:16.526654 17012 raft_consensus.cc:3037] T bd10315524d7434a88b3bb4b2a227952 P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:16.527616 16880 raft_consensus.cc:2454] T bd10315524d7434a88b3bb4b2a227952 P e1a9c39d088a43e7b6f9beee2068deeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 626e808943c54503bc4bd8cf1cd3e4b5 in term 1.
I20240501 14:16:16.527630 17012 raft_consensus.cc:2454] T bd10315524d7434a88b3bb4b2a227952 P 48789b208dcf4f8ab388470fa43a751b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 626e808943c54503bc4bd8cf1cd3e4b5 in term 1.
I20240501 14:16:16.527897 16683 leader_election.cc:302] T bd10315524d7434a88b3bb4b2a227952 P 626e808943c54503bc4bd8cf1cd3e4b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48789b208dcf4f8ab388470fa43a751b, 626e808943c54503bc4bd8cf1cd3e4b5; no voters:
I20240501 14:16:16.528055 17219 raft_consensus.cc:2781] T bd10315524d7434a88b3bb4b2a227952 P 626e808943c54503bc4bd8cf1cd3e4b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:16:16.528136 17219 raft_consensus.cc:686] T bd10315524d7434a88b3bb4b2a227952 P 626e808943c54503bc4bd8cf1cd3e4b5 [term 1 LEADER]: Becoming Leader. State: Replica: 626e808943c54503bc4bd8cf1cd3e4b5, State: Running, Role: LEADER
I20240501 14:16:16.528286 17219 consensus_queue.cc:237] T bd10315524d7434a88b3bb4b2a227952 P 626e808943c54503bc4bd8cf1cd3e4b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.529943 16606 catalog_manager.cc:5484] T bd10315524d7434a88b3bb4b2a227952 P 626e808943c54503bc4bd8cf1cd3e4b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 626e808943c54503bc4bd8cf1cd3e4b5 (127.7.39.129). New cstate: current_term: 1 leader_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } health_report { overall_health: UNKNOWN } } }
W20240501 14:16:16.670150 17303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:16:16.670411 17303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:16:16.674229 17303 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 14:16:16.705719 17268 consensus_queue.cc:1035] T e82af677f1f644549a3213b4b8bf3d4f P 48789b208dcf4f8ab388470fa43a751b [LEADER]: Connected to new peer: Peer: permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:16:16.710427 17268 consensus_queue.cc:1035] T e82af677f1f644549a3213b4b8bf3d4f P 48789b208dcf4f8ab388470fa43a751b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:16:16.773463 17012 raft_consensus.cc:1261] T 5639d52050b8423393104b34c9d970ac P 48789b208dcf4f8ab388470fa43a751b [term 1 FOLLOWER]: Refusing update from remote peer e1a9c39d088a43e7b6f9beee2068deeb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:16:16.774036 17300 consensus_queue.cc:1035] T 5639d52050b8423393104b34c9d970ac P e1a9c39d088a43e7b6f9beee2068deeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:16:16.775131 17011 raft_consensus.cc:1261] T bd10315524d7434a88b3bb4b2a227952 P 48789b208dcf4f8ab388470fa43a751b [term 1 FOLLOWER]: Refusing update from remote peer 626e808943c54503bc4bd8cf1cd3e4b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:16:16.775166 16880 raft_consensus.cc:1261] T bd10315524d7434a88b3bb4b2a227952 P e1a9c39d088a43e7b6f9beee2068deeb [term 1 FOLLOWER]: Refusing update from remote peer 626e808943c54503bc4bd8cf1cd3e4b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:16:16.775635 17219 consensus_queue.cc:1035] T bd10315524d7434a88b3bb4b2a227952 P 626e808943c54503bc4bd8cf1cd3e4b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:16:16.776165 16748 raft_consensus.cc:1261] T 5639d52050b8423393104b34c9d970ac P 626e808943c54503bc4bd8cf1cd3e4b5 [term 1 FOLLOWER]: Refusing update from remote peer e1a9c39d088a43e7b6f9beee2068deeb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:16:16.776314 17219 consensus_queue.cc:1035] T bd10315524d7434a88b3bb4b2a227952 P 626e808943c54503bc4bd8cf1cd3e4b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:16:16.776600 17300 consensus_queue.cc:1035] T 5639d52050b8423393104b34c9d970ac P e1a9c39d088a43e7b6f9beee2068deeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:16:16.792567 17328 mvcc.cc:205] Tried to move back new op lower bound from 7022890912858648576 to 7022890911524167680. Current Snapshot: MvccSnapshot[applied={T|T < 7022890912858648576}]
INSERT report
rows total: 2000
time total: 127.01 ms
time per row: 0.0635052 ms
I20240501 14:16:16.841619 16597 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36462:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240501 14:16:16.842131 16597 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 14:16:16.847016 16858 tablet_service.cc:1467] Processing CreateTablet for tablet 3c4bc2f0d8ec4183b4edb54cdbcac1cc (DEFAULT_TABLE table=rebalance_test_table_2 [id=c23e70d314a647f0bff92364d67255b3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:16.847134 16726 tablet_service.cc:1467] Processing CreateTablet for tablet 3c4bc2f0d8ec4183b4edb54cdbcac1cc (DEFAULT_TABLE table=rebalance_test_table_2 [id=c23e70d314a647f0bff92364d67255b3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:16.847389 16858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c4bc2f0d8ec4183b4edb54cdbcac1cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:16.847431 16726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c4bc2f0d8ec4183b4edb54cdbcac1cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:16.847414 16990 tablet_service.cc:1467] Processing CreateTablet for tablet 3c4bc2f0d8ec4183b4edb54cdbcac1cc (DEFAULT_TABLE table=rebalance_test_table_2 [id=c23e70d314a647f0bff92364d67255b3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:16.847677 16990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c4bc2f0d8ec4183b4edb54cdbcac1cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:16.849207 17361 tablet_bootstrap.cc:492] T 3c4bc2f0d8ec4183b4edb54cdbcac1cc P e1a9c39d088a43e7b6f9beee2068deeb: Bootstrap starting.
I20240501 14:16:16.847134 16728 tablet_service.cc:1467] Processing CreateTablet for tablet 76362c2867ba403f9b9011f6b2247137 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c23e70d314a647f0bff92364d67255b3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:16.849782 16728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76362c2867ba403f9b9011f6b2247137. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:16.850133 17361 tablet_bootstrap.cc:654] T 3c4bc2f0d8ec4183b4edb54cdbcac1cc P e1a9c39d088a43e7b6f9beee2068deeb: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:16.850281 16858 tablet_service.cc:1467] Processing CreateTablet for tablet 76362c2867ba403f9b9011f6b2247137 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c23e70d314a647f0bff92364d67255b3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:16.850443 17363 tablet_bootstrap.cc:492] T 3c4bc2f0d8ec4183b4edb54cdbcac1cc P 48789b208dcf4f8ab388470fa43a751b: Bootstrap starting.
I20240501 14:16:16.850536 16858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76362c2867ba403f9b9011f6b2247137. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:16.851332 17363 tablet_bootstrap.cc:654] T 3c4bc2f0d8ec4183b4edb54cdbcac1cc P 48789b208dcf4f8ab388470fa43a751b: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:16.851814 17362 tablet_bootstrap.cc:492] T 3c4bc2f0d8ec4183b4edb54cdbcac1cc P 626e808943c54503bc4bd8cf1cd3e4b5: Bootstrap starting.
I20240501 14:16:16.850281 16860 tablet_service.cc:1467] Processing CreateTablet for tablet 88530fdb379743cf9ab3d1831d4e8d9c (DEFAULT_TABLE table=rebalance_test_table_2 [id=c23e70d314a647f0bff92364d67255b3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:16.852152 16860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88530fdb379743cf9ab3d1831d4e8d9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:16.852118 16728 tablet_service.cc:1467] Processing CreateTablet for tablet 88530fdb379743cf9ab3d1831d4e8d9c (DEFAULT_TABLE table=rebalance_test_table_2 [id=c23e70d314a647f0bff92364d67255b3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:16.852332 16728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88530fdb379743cf9ab3d1831d4e8d9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:16.852947 17362 tablet_bootstrap.cc:654] T 3c4bc2f0d8ec4183b4edb54cdbcac1cc P 626e808943c54503bc4bd8cf1cd3e4b5: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:16.853935 16992 tablet_service.cc:1467] Processing CreateTablet for tablet 88530fdb379743cf9ab3d1831d4e8d9c (DEFAULT_TABLE table=rebalance_test_table_2 [id=c23e70d314a647f0bff92364d67255b3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:16.854180 16992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88530fdb379743cf9ab3d1831d4e8d9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:16.854593 17362 tablet_bootstrap.cc:492] T 3c4bc2f0d8ec4183b4edb54cdbcac1cc P 626e808943c54503bc4bd8cf1cd3e4b5: No bootstrap required, opened a new log
I20240501 14:16:16.854678 17362 ts_tablet_manager.cc:1367] T 3c4bc2f0d8ec4183b4edb54cdbcac1cc P 626e808943c54503bc4bd8cf1cd3e4b5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:16:16.854974 17363 tablet_bootstrap.cc:492] T 3c4bc2f0d8ec4183b4edb54cdbcac1cc P 48789b208dcf4f8ab388470fa43a751b: No bootstrap required, opened a new log
I20240501 14:16:16.855110 17363 ts_tablet_manager.cc:1367] T 3c4bc2f0d8ec4183b4edb54cdbcac1cc P 48789b208dcf4f8ab388470fa43a751b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:16:16.855172 17362 raft_consensus.cc:348] T 3c4bc2f0d8ec4183b4edb54cdbcac1cc P 626e808943c54503bc4bd8cf1cd3e4b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.855311 17362 raft_consensus.cc:374] T 3c4bc2f0d8ec4183b4edb54cdbcac1cc P 626e808943c54503bc4bd8cf1cd3e4b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:16.855358 17362 raft_consensus.cc:729] T 3c4bc2f0d8ec4183b4edb54cdbcac1cc P 626e808943c54503bc4bd8cf1cd3e4b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 626e808943c54503bc4bd8cf1cd3e4b5, State: Initialized, Role: FOLLOWER
I20240501 14:16:16.855504 17362 consensus_queue.cc:260] T 3c4bc2f0d8ec4183b4edb54cdbcac1cc P 626e808943c54503bc4bd8cf1cd3e4b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.855583 17363 raft_consensus.cc:348] T 3c4bc2f0d8ec4183b4edb54cdbcac1cc P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.855707 17363 raft_consensus.cc:374] T 3c4bc2f0d8ec4183b4edb54cdbcac1cc P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:16.855751 17363 raft_consensus.cc:729] T 3c4bc2f0d8ec4183b4edb54cdbcac1cc P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48789b208dcf4f8ab388470fa43a751b, State: Initialized, Role: FOLLOWER
I20240501 14:16:16.855780 17362 ts_tablet_manager.cc:1398] T 3c4bc2f0d8ec4183b4edb54cdbcac1cc P 626e808943c54503bc4bd8cf1cd3e4b5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:16:16.855913 17363 consensus_queue.cc:260] T 3c4bc2f0d8ec4183b4edb54cdbcac1cc P 48789b208dcf4f8ab388470fa43a751b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.855962 17362 tablet_bootstrap.cc:492] T 76362c2867ba403f9b9011f6b2247137 P 626e808943c54503bc4bd8cf1cd3e4b5: Bootstrap starting.
I20240501 14:16:16.856539 16990 tablet_service.cc:1467] Processing CreateTablet for tablet 76362c2867ba403f9b9011f6b2247137 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c23e70d314a647f0bff92364d67255b3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:16:16.856755 16990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76362c2867ba403f9b9011f6b2247137. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:16:16.856791 17362 tablet_bootstrap.cc:654] T 76362c2867ba403f9b9011f6b2247137 P 626e808943c54503bc4bd8cf1cd3e4b5: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:16.856982 17363 ts_tablet_manager.cc:1398] T 3c4bc2f0d8ec4183b4edb54cdbcac1cc P 48789b208dcf4f8ab388470fa43a751b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:16:16.857164 17363 tablet_bootstrap.cc:492] T 88530fdb379743cf9ab3d1831d4e8d9c P 48789b208dcf4f8ab388470fa43a751b: Bootstrap starting.
I20240501 14:16:16.858036 17363 tablet_bootstrap.cc:654] T 88530fdb379743cf9ab3d1831d4e8d9c P 48789b208dcf4f8ab388470fa43a751b: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:16.859238 17363 tablet_bootstrap.cc:492] T 88530fdb379743cf9ab3d1831d4e8d9c P 48789b208dcf4f8ab388470fa43a751b: No bootstrap required, opened a new log
I20240501 14:16:16.859330 17363 ts_tablet_manager.cc:1367] T 88530fdb379743cf9ab3d1831d4e8d9c P 48789b208dcf4f8ab388470fa43a751b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:16:16.859781 17361 tablet_bootstrap.cc:492] T 3c4bc2f0d8ec4183b4edb54cdbcac1cc P e1a9c39d088a43e7b6f9beee2068deeb: No bootstrap required, opened a new log
I20240501 14:16:16.859750 17363 raft_consensus.cc:348] T 88530fdb379743cf9ab3d1831d4e8d9c P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.859850 17363 raft_consensus.cc:374] T 88530fdb379743cf9ab3d1831d4e8d9c P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:16.859891 17361 ts_tablet_manager.cc:1367] T 3c4bc2f0d8ec4183b4edb54cdbcac1cc P e1a9c39d088a43e7b6f9beee2068deeb: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 14:16:16.859892 17363 raft_consensus.cc:729] T 88530fdb379743cf9ab3d1831d4e8d9c P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48789b208dcf4f8ab388470fa43a751b, State: Initialized, Role: FOLLOWER
I20240501 14:16:16.860025 17363 consensus_queue.cc:260] T 88530fdb379743cf9ab3d1831d4e8d9c P 48789b208dcf4f8ab388470fa43a751b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.860373 17363 ts_tablet_manager.cc:1398] T 88530fdb379743cf9ab3d1831d4e8d9c P 48789b208dcf4f8ab388470fa43a751b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:16:16.860425 17361 raft_consensus.cc:348] T 3c4bc2f0d8ec4183b4edb54cdbcac1cc P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.860527 17361 raft_consensus.cc:374] T 3c4bc2f0d8ec4183b4edb54cdbcac1cc P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:16.860531 17363 tablet_bootstrap.cc:492] T 76362c2867ba403f9b9011f6b2247137 P 48789b208dcf4f8ab388470fa43a751b: Bootstrap starting.
I20240501 14:16:16.860561 17361 raft_consensus.cc:729] T 3c4bc2f0d8ec4183b4edb54cdbcac1cc P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a9c39d088a43e7b6f9beee2068deeb, State: Initialized, Role: FOLLOWER
I20240501 14:16:16.860684 17361 consensus_queue.cc:260] T 3c4bc2f0d8ec4183b4edb54cdbcac1cc P e1a9c39d088a43e7b6f9beee2068deeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.860966 17361 ts_tablet_manager.cc:1398] T 3c4bc2f0d8ec4183b4edb54cdbcac1cc P e1a9c39d088a43e7b6f9beee2068deeb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:16:16.861174 17361 tablet_bootstrap.cc:492] T 88530fdb379743cf9ab3d1831d4e8d9c P e1a9c39d088a43e7b6f9beee2068deeb: Bootstrap starting.
I20240501 14:16:16.861347 17363 tablet_bootstrap.cc:654] T 76362c2867ba403f9b9011f6b2247137 P 48789b208dcf4f8ab388470fa43a751b: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:16.861440 17362 tablet_bootstrap.cc:492] T 76362c2867ba403f9b9011f6b2247137 P 626e808943c54503bc4bd8cf1cd3e4b5: No bootstrap required, opened a new log
I20240501 14:16:16.861515 17362 ts_tablet_manager.cc:1367] T 76362c2867ba403f9b9011f6b2247137 P 626e808943c54503bc4bd8cf1cd3e4b5: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240501 14:16:16.862059 17361 tablet_bootstrap.cc:654] T 88530fdb379743cf9ab3d1831d4e8d9c P e1a9c39d088a43e7b6f9beee2068deeb: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:16.862385 17363 tablet_bootstrap.cc:492] T 76362c2867ba403f9b9011f6b2247137 P 48789b208dcf4f8ab388470fa43a751b: No bootstrap required, opened a new log
I20240501 14:16:16.862459 17363 ts_tablet_manager.cc:1367] T 76362c2867ba403f9b9011f6b2247137 P 48789b208dcf4f8ab388470fa43a751b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:16:16.862656 17362 raft_consensus.cc:348] T 76362c2867ba403f9b9011f6b2247137 P 626e808943c54503bc4bd8cf1cd3e4b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.862731 17362 raft_consensus.cc:374] T 76362c2867ba403f9b9011f6b2247137 P 626e808943c54503bc4bd8cf1cd3e4b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:16.862751 17362 raft_consensus.cc:729] T 76362c2867ba403f9b9011f6b2247137 P 626e808943c54503bc4bd8cf1cd3e4b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 626e808943c54503bc4bd8cf1cd3e4b5, State: Initialized, Role: FOLLOWER
I20240501 14:16:16.862833 17362 consensus_queue.cc:260] T 76362c2867ba403f9b9011f6b2247137 P 626e808943c54503bc4bd8cf1cd3e4b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.862835 17363 raft_consensus.cc:348] T 76362c2867ba403f9b9011f6b2247137 P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.862934 17363 raft_consensus.cc:374] T 76362c2867ba403f9b9011f6b2247137 P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:16.862967 17363 raft_consensus.cc:729] T 76362c2867ba403f9b9011f6b2247137 P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48789b208dcf4f8ab388470fa43a751b, State: Initialized, Role: FOLLOWER
I20240501 14:16:16.863061 17362 ts_tablet_manager.cc:1398] T 76362c2867ba403f9b9011f6b2247137 P 626e808943c54503bc4bd8cf1cd3e4b5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:16:16.863108 17363 consensus_queue.cc:260] T 76362c2867ba403f9b9011f6b2247137 P 48789b208dcf4f8ab388470fa43a751b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.863198 17362 tablet_bootstrap.cc:492] T 88530fdb379743cf9ab3d1831d4e8d9c P 626e808943c54503bc4bd8cf1cd3e4b5: Bootstrap starting.
I20240501 14:16:16.863368 17363 ts_tablet_manager.cc:1398] T 76362c2867ba403f9b9011f6b2247137 P 48789b208dcf4f8ab388470fa43a751b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:16:16.863871 17362 tablet_bootstrap.cc:654] T 88530fdb379743cf9ab3d1831d4e8d9c P 626e808943c54503bc4bd8cf1cd3e4b5: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:16.864113 17361 tablet_bootstrap.cc:492] T 88530fdb379743cf9ab3d1831d4e8d9c P e1a9c39d088a43e7b6f9beee2068deeb: No bootstrap required, opened a new log
I20240501 14:16:16.864207 17361 ts_tablet_manager.cc:1367] T 88530fdb379743cf9ab3d1831d4e8d9c P e1a9c39d088a43e7b6f9beee2068deeb: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:16:16.864675 17361 raft_consensus.cc:348] T 88530fdb379743cf9ab3d1831d4e8d9c P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.864786 17361 raft_consensus.cc:374] T 88530fdb379743cf9ab3d1831d4e8d9c P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:16.864830 17361 raft_consensus.cc:729] T 88530fdb379743cf9ab3d1831d4e8d9c P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a9c39d088a43e7b6f9beee2068deeb, State: Initialized, Role: FOLLOWER
I20240501 14:16:16.864940 17362 tablet_bootstrap.cc:492] T 88530fdb379743cf9ab3d1831d4e8d9c P 626e808943c54503bc4bd8cf1cd3e4b5: No bootstrap required, opened a new log
I20240501 14:16:16.864951 17361 consensus_queue.cc:260] T 88530fdb379743cf9ab3d1831d4e8d9c P e1a9c39d088a43e7b6f9beee2068deeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.865020 17362 ts_tablet_manager.cc:1367] T 88530fdb379743cf9ab3d1831d4e8d9c P 626e808943c54503bc4bd8cf1cd3e4b5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:16:16.865234 17361 ts_tablet_manager.cc:1398] T 88530fdb379743cf9ab3d1831d4e8d9c P e1a9c39d088a43e7b6f9beee2068deeb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:16:16.865455 17361 tablet_bootstrap.cc:492] T 76362c2867ba403f9b9011f6b2247137 P e1a9c39d088a43e7b6f9beee2068deeb: Bootstrap starting.
I20240501 14:16:16.865506 17362 raft_consensus.cc:348] T 88530fdb379743cf9ab3d1831d4e8d9c P 626e808943c54503bc4bd8cf1cd3e4b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.865608 17362 raft_consensus.cc:374] T 88530fdb379743cf9ab3d1831d4e8d9c P 626e808943c54503bc4bd8cf1cd3e4b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:16.865639 17362 raft_consensus.cc:729] T 88530fdb379743cf9ab3d1831d4e8d9c P 626e808943c54503bc4bd8cf1cd3e4b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 626e808943c54503bc4bd8cf1cd3e4b5, State: Initialized, Role: FOLLOWER
I20240501 14:16:16.865744 17362 consensus_queue.cc:260] T 88530fdb379743cf9ab3d1831d4e8d9c P 626e808943c54503bc4bd8cf1cd3e4b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.866006 17362 ts_tablet_manager.cc:1398] T 88530fdb379743cf9ab3d1831d4e8d9c P 626e808943c54503bc4bd8cf1cd3e4b5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:16:16.866359 17361 tablet_bootstrap.cc:654] T 76362c2867ba403f9b9011f6b2247137 P e1a9c39d088a43e7b6f9beee2068deeb: Neither blocks nor log segments found. Creating new log.
I20240501 14:16:16.867612 17361 tablet_bootstrap.cc:492] T 76362c2867ba403f9b9011f6b2247137 P e1a9c39d088a43e7b6f9beee2068deeb: No bootstrap required, opened a new log
I20240501 14:16:16.867699 17361 ts_tablet_manager.cc:1367] T 76362c2867ba403f9b9011f6b2247137 P e1a9c39d088a43e7b6f9beee2068deeb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:16:16.868208 17361 raft_consensus.cc:348] T 76362c2867ba403f9b9011f6b2247137 P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.868333 17361 raft_consensus.cc:374] T 76362c2867ba403f9b9011f6b2247137 P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:16:16.868363 17361 raft_consensus.cc:729] T 76362c2867ba403f9b9011f6b2247137 P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a9c39d088a43e7b6f9beee2068deeb, State: Initialized, Role: FOLLOWER
I20240501 14:16:16.868479 17361 consensus_queue.cc:260] T 76362c2867ba403f9b9011f6b2247137 P e1a9c39d088a43e7b6f9beee2068deeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.868726 17361 ts_tablet_manager.cc:1398] T 76362c2867ba403f9b9011f6b2247137 P e1a9c39d088a43e7b6f9beee2068deeb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:16:16.914965 17215 raft_consensus.cc:482] T 88530fdb379743cf9ab3d1831d4e8d9c P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:16:16.915129 17215 raft_consensus.cc:504] T 88530fdb379743cf9ab3d1831d4e8d9c P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.915527 17215 leader_election.cc:288] T 88530fdb379743cf9ab3d1831d4e8d9c P e1a9c39d088a43e7b6f9beee2068deeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 626e808943c54503bc4bd8cf1cd3e4b5 (127.7.39.129:43915), 48789b208dcf4f8ab388470fa43a751b (127.7.39.131:45011)
I20240501 14:16:16.916029 17011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88530fdb379743cf9ab3d1831d4e8d9c" candidate_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48789b208dcf4f8ab388470fa43a751b" is_pre_election: true
I20240501 14:16:16.916045 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88530fdb379743cf9ab3d1831d4e8d9c" candidate_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" is_pre_election: true
I20240501 14:16:16.916174 17011 raft_consensus.cc:2454] T 88530fdb379743cf9ab3d1831d4e8d9c P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1a9c39d088a43e7b6f9beee2068deeb in term 0.
I20240501 14:16:16.916205 16748 raft_consensus.cc:2454] T 88530fdb379743cf9ab3d1831d4e8d9c P 626e808943c54503bc4bd8cf1cd3e4b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1a9c39d088a43e7b6f9beee2068deeb in term 0.
I20240501 14:16:16.916507 16815 leader_election.cc:302] T 88530fdb379743cf9ab3d1831d4e8d9c P e1a9c39d088a43e7b6f9beee2068deeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 626e808943c54503bc4bd8cf1cd3e4b5, e1a9c39d088a43e7b6f9beee2068deeb; no voters:
I20240501 14:16:16.916812 17215 raft_consensus.cc:2781] T 88530fdb379743cf9ab3d1831d4e8d9c P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:16:16.916862 17215 raft_consensus.cc:482] T 88530fdb379743cf9ab3d1831d4e8d9c P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:16:16.916904 17215 raft_consensus.cc:3037] T 88530fdb379743cf9ab3d1831d4e8d9c P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:16.917762 17215 raft_consensus.cc:504] T 88530fdb379743cf9ab3d1831d4e8d9c P e1a9c39d088a43e7b6f9beee2068deeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.918072 17215 leader_election.cc:288] T 88530fdb379743cf9ab3d1831d4e8d9c P e1a9c39d088a43e7b6f9beee2068deeb [CANDIDATE]: Term 1 election: Requested vote from peers 626e808943c54503bc4bd8cf1cd3e4b5 (127.7.39.129:43915), 48789b208dcf4f8ab388470fa43a751b (127.7.39.131:45011)
I20240501 14:16:16.918416 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88530fdb379743cf9ab3d1831d4e8d9c" candidate_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "626e808943c54503bc4bd8cf1cd3e4b5"
I20240501 14:16:16.918479 17011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88530fdb379743cf9ab3d1831d4e8d9c" candidate_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48789b208dcf4f8ab388470fa43a751b"
I20240501 14:16:16.918545 16748 raft_consensus.cc:3037] T 88530fdb379743cf9ab3d1831d4e8d9c P 626e808943c54503bc4bd8cf1cd3e4b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:16.918582 17011 raft_consensus.cc:3037] T 88530fdb379743cf9ab3d1831d4e8d9c P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:16.919559 16748 raft_consensus.cc:2454] T 88530fdb379743cf9ab3d1831d4e8d9c P 626e808943c54503bc4bd8cf1cd3e4b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1a9c39d088a43e7b6f9beee2068deeb in term 1.
I20240501 14:16:16.919564 17011 raft_consensus.cc:2454] T 88530fdb379743cf9ab3d1831d4e8d9c P 48789b208dcf4f8ab388470fa43a751b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1a9c39d088a43e7b6f9beee2068deeb in term 1.
I20240501 14:16:16.919852 16815 leader_election.cc:302] T 88530fdb379743cf9ab3d1831d4e8d9c P e1a9c39d088a43e7b6f9beee2068deeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48789b208dcf4f8ab388470fa43a751b, e1a9c39d088a43e7b6f9beee2068deeb; no voters:
I20240501 14:16:16.920078 17215 raft_consensus.cc:2781] T 88530fdb379743cf9ab3d1831d4e8d9c P e1a9c39d088a43e7b6f9beee2068deeb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:16:16.920146 17215 raft_consensus.cc:686] T 88530fdb379743cf9ab3d1831d4e8d9c P e1a9c39d088a43e7b6f9beee2068deeb [term 1 LEADER]: Becoming Leader. State: Replica: e1a9c39d088a43e7b6f9beee2068deeb, State: Running, Role: LEADER
I20240501 14:16:16.920289 17215 consensus_queue.cc:237] T 88530fdb379743cf9ab3d1831d4e8d9c P e1a9c39d088a43e7b6f9beee2068deeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.921947 16597 catalog_manager.cc:5484] T 88530fdb379743cf9ab3d1831d4e8d9c P e1a9c39d088a43e7b6f9beee2068deeb reported cstate change: term changed from 0 to 1, leader changed from <none> to e1a9c39d088a43e7b6f9beee2068deeb (127.7.39.130). New cstate: current_term: 1 leader_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } health_report { overall_health: UNKNOWN } } }
I20240501 14:16:16.924451 17339 raft_consensus.cc:482] T 3c4bc2f0d8ec4183b4edb54cdbcac1cc P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:16:16.924546 17339 raft_consensus.cc:504] T 3c4bc2f0d8ec4183b4edb54cdbcac1cc P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.924934 17339 leader_election.cc:288] T 3c4bc2f0d8ec4183b4edb54cdbcac1cc P 48789b208dcf4f8ab388470fa43a751b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1a9c39d088a43e7b6f9beee2068deeb (127.7.39.130:40479), 626e808943c54503bc4bd8cf1cd3e4b5 (127.7.39.129:43915)
I20240501 14:16:16.925277 16880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4bc2f0d8ec4183b4edb54cdbcac1cc" candidate_uuid: "48789b208dcf4f8ab388470fa43a751b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" is_pre_election: true
I20240501 14:16:16.925348 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4bc2f0d8ec4183b4edb54cdbcac1cc" candidate_uuid: "48789b208dcf4f8ab388470fa43a751b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" is_pre_election: true
I20240501 14:16:16.925433 16880 raft_consensus.cc:2454] T 3c4bc2f0d8ec4183b4edb54cdbcac1cc P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48789b208dcf4f8ab388470fa43a751b in term 0.
I20240501 14:16:16.925470 16748 raft_consensus.cc:2454] T 3c4bc2f0d8ec4183b4edb54cdbcac1cc P 626e808943c54503bc4bd8cf1cd3e4b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48789b208dcf4f8ab388470fa43a751b in term 0.
I20240501 14:16:16.925758 16948 leader_election.cc:302] T 3c4bc2f0d8ec4183b4edb54cdbcac1cc P 48789b208dcf4f8ab388470fa43a751b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48789b208dcf4f8ab388470fa43a751b, e1a9c39d088a43e7b6f9beee2068deeb; no voters:
I20240501 14:16:16.926110 17339 raft_consensus.cc:2781] T 3c4bc2f0d8ec4183b4edb54cdbcac1cc P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:16:16.926172 17339 raft_consensus.cc:482] T 3c4bc2f0d8ec4183b4edb54cdbcac1cc P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:16:16.926229 17339 raft_consensus.cc:3037] T 3c4bc2f0d8ec4183b4edb54cdbcac1cc P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:16.926929 17339 raft_consensus.cc:504] T 3c4bc2f0d8ec4183b4edb54cdbcac1cc P 48789b208dcf4f8ab388470fa43a751b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.927258 17339 leader_election.cc:288] T 3c4bc2f0d8ec4183b4edb54cdbcac1cc P 48789b208dcf4f8ab388470fa43a751b [CANDIDATE]: Term 1 election: Requested vote from peers e1a9c39d088a43e7b6f9beee2068deeb (127.7.39.130:40479), 626e808943c54503bc4bd8cf1cd3e4b5 (127.7.39.129:43915)
I20240501 14:16:16.927558 16880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4bc2f0d8ec4183b4edb54cdbcac1cc" candidate_uuid: "48789b208dcf4f8ab388470fa43a751b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9c39d088a43e7b6f9beee2068deeb"
I20240501 14:16:16.927585 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4bc2f0d8ec4183b4edb54cdbcac1cc" candidate_uuid: "48789b208dcf4f8ab388470fa43a751b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "626e808943c54503bc4bd8cf1cd3e4b5"
I20240501 14:16:16.927666 16880 raft_consensus.cc:3037] T 3c4bc2f0d8ec4183b4edb54cdbcac1cc P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:16.927678 16748 raft_consensus.cc:3037] T 3c4bc2f0d8ec4183b4edb54cdbcac1cc P 626e808943c54503bc4bd8cf1cd3e4b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:16.928365 16880 raft_consensus.cc:2454] T 3c4bc2f0d8ec4183b4edb54cdbcac1cc P e1a9c39d088a43e7b6f9beee2068deeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48789b208dcf4f8ab388470fa43a751b in term 1.
I20240501 14:16:16.928372 16748 raft_consensus.cc:2454] T 3c4bc2f0d8ec4183b4edb54cdbcac1cc P 626e808943c54503bc4bd8cf1cd3e4b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48789b208dcf4f8ab388470fa43a751b in term 1.
I20240501 14:16:16.928624 16948 leader_election.cc:302] T 3c4bc2f0d8ec4183b4edb54cdbcac1cc P 48789b208dcf4f8ab388470fa43a751b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48789b208dcf4f8ab388470fa43a751b, e1a9c39d088a43e7b6f9beee2068deeb; no voters:
I20240501 14:16:16.928833 17339 raft_consensus.cc:2781] T 3c4bc2f0d8ec4183b4edb54cdbcac1cc P 48789b208dcf4f8ab388470fa43a751b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:16:16.928946 17339 raft_consensus.cc:686] T 3c4bc2f0d8ec4183b4edb54cdbcac1cc P 48789b208dcf4f8ab388470fa43a751b [term 1 LEADER]: Becoming Leader. State: Replica: 48789b208dcf4f8ab388470fa43a751b, State: Running, Role: LEADER
I20240501 14:16:16.929107 17339 consensus_queue.cc:237] T 3c4bc2f0d8ec4183b4edb54cdbcac1cc P 48789b208dcf4f8ab388470fa43a751b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.930670 16597 catalog_manager.cc:5484] T 3c4bc2f0d8ec4183b4edb54cdbcac1cc P 48789b208dcf4f8ab388470fa43a751b reported cstate change: term changed from 0 to 1, leader changed from <none> to 48789b208dcf4f8ab388470fa43a751b (127.7.39.131). New cstate: current_term: 1 leader_uuid: "48789b208dcf4f8ab388470fa43a751b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } health_report { overall_health: HEALTHY } } }
I20240501 14:16:16.934911 17215 raft_consensus.cc:482] T 76362c2867ba403f9b9011f6b2247137 P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:16:16.934999 17215 raft_consensus.cc:504] T 76362c2867ba403f9b9011f6b2247137 P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.935338 17215 leader_election.cc:288] T 76362c2867ba403f9b9011f6b2247137 P e1a9c39d088a43e7b6f9beee2068deeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 626e808943c54503bc4bd8cf1cd3e4b5 (127.7.39.129:43915), 48789b208dcf4f8ab388470fa43a751b (127.7.39.131:45011)
I20240501 14:16:16.935638 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76362c2867ba403f9b9011f6b2247137" candidate_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" is_pre_election: true
I20240501 14:16:16.935667 17011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76362c2867ba403f9b9011f6b2247137" candidate_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48789b208dcf4f8ab388470fa43a751b" is_pre_election: true
I20240501 14:16:16.935765 16748 raft_consensus.cc:2454] T 76362c2867ba403f9b9011f6b2247137 P 626e808943c54503bc4bd8cf1cd3e4b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1a9c39d088a43e7b6f9beee2068deeb in term 0.
I20240501 14:16:16.935806 17011 raft_consensus.cc:2454] T 76362c2867ba403f9b9011f6b2247137 P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1a9c39d088a43e7b6f9beee2068deeb in term 0.
I20240501 14:16:16.936072 16815 leader_election.cc:302] T 76362c2867ba403f9b9011f6b2247137 P e1a9c39d088a43e7b6f9beee2068deeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 626e808943c54503bc4bd8cf1cd3e4b5, e1a9c39d088a43e7b6f9beee2068deeb; no voters:
I20240501 14:16:16.936290 17215 raft_consensus.cc:2781] T 76362c2867ba403f9b9011f6b2247137 P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:16:16.936362 17215 raft_consensus.cc:482] T 76362c2867ba403f9b9011f6b2247137 P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:16:16.936398 17215 raft_consensus.cc:3037] T 76362c2867ba403f9b9011f6b2247137 P e1a9c39d088a43e7b6f9beee2068deeb [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:16.937261 17215 raft_consensus.cc:504] T 76362c2867ba403f9b9011f6b2247137 P e1a9c39d088a43e7b6f9beee2068deeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.937610 17215 leader_election.cc:288] T 76362c2867ba403f9b9011f6b2247137 P e1a9c39d088a43e7b6f9beee2068deeb [CANDIDATE]: Term 1 election: Requested vote from peers 626e808943c54503bc4bd8cf1cd3e4b5 (127.7.39.129:43915), 48789b208dcf4f8ab388470fa43a751b (127.7.39.131:45011)
I20240501 14:16:16.937934 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76362c2867ba403f9b9011f6b2247137" candidate_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "626e808943c54503bc4bd8cf1cd3e4b5"
I20240501 14:16:16.937985 17011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76362c2867ba403f9b9011f6b2247137" candidate_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48789b208dcf4f8ab388470fa43a751b"
I20240501 14:16:16.938055 16748 raft_consensus.cc:3037] T 76362c2867ba403f9b9011f6b2247137 P 626e808943c54503bc4bd8cf1cd3e4b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:16.938079 17011 raft_consensus.cc:3037] T 76362c2867ba403f9b9011f6b2247137 P 48789b208dcf4f8ab388470fa43a751b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:16:16.938939 16748 raft_consensus.cc:2454] T 76362c2867ba403f9b9011f6b2247137 P 626e808943c54503bc4bd8cf1cd3e4b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1a9c39d088a43e7b6f9beee2068deeb in term 1.
I20240501 14:16:16.938939 17011 raft_consensus.cc:2454] T 76362c2867ba403f9b9011f6b2247137 P 48789b208dcf4f8ab388470fa43a751b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1a9c39d088a43e7b6f9beee2068deeb in term 1.
I20240501 14:16:16.939224 16815 leader_election.cc:302] T 76362c2867ba403f9b9011f6b2247137 P e1a9c39d088a43e7b6f9beee2068deeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48789b208dcf4f8ab388470fa43a751b, e1a9c39d088a43e7b6f9beee2068deeb; no voters:
I20240501 14:16:16.939369 17215 raft_consensus.cc:2781] T 76362c2867ba403f9b9011f6b2247137 P e1a9c39d088a43e7b6f9beee2068deeb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:16:16.939457 17215 raft_consensus.cc:686] T 76362c2867ba403f9b9011f6b2247137 P e1a9c39d088a43e7b6f9beee2068deeb [term 1 LEADER]: Becoming Leader. State: Replica: e1a9c39d088a43e7b6f9beee2068deeb, State: Running, Role: LEADER
I20240501 14:16:16.939590 17215 consensus_queue.cc:237] T 76362c2867ba403f9b9011f6b2247137 P e1a9c39d088a43e7b6f9beee2068deeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } } peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } }
I20240501 14:16:16.941110 16600 catalog_manager.cc:5484] T 76362c2867ba403f9b9011f6b2247137 P e1a9c39d088a43e7b6f9beee2068deeb reported cstate change: term changed from 0 to 1, leader changed from <none> to e1a9c39d088a43e7b6f9beee2068deeb (127.7.39.130). New cstate: current_term: 1 leader_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 } health_report { overall_health: UNKNOWN } } }
W20240501 14:16:17.055012 17376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:16:17.055279 17376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:16:17.058147 17376 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240501 14:16:17.131938 17011 raft_consensus.cc:1261] T 76362c2867ba403f9b9011f6b2247137 P 48789b208dcf4f8ab388470fa43a751b [term 1 FOLLOWER]: Refusing update from remote peer e1a9c39d088a43e7b6f9beee2068deeb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:16:17.132547 16748 raft_consensus.cc:1261] T 3c4bc2f0d8ec4183b4edb54cdbcac1cc P 626e808943c54503bc4bd8cf1cd3e4b5 [term 1 FOLLOWER]: Refusing update from remote peer 48789b208dcf4f8ab388470fa43a751b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:16:17.132916 16747 raft_consensus.cc:1261] T 76362c2867ba403f9b9011f6b2247137 P 626e808943c54503bc4bd8cf1cd3e4b5 [term 1 FOLLOWER]: Refusing update from remote peer e1a9c39d088a43e7b6f9beee2068deeb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:16:17.133294 17300 consensus_queue.cc:1035] T 76362c2867ba403f9b9011f6b2247137 P e1a9c39d088a43e7b6f9beee2068deeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:16:17.133371 17339 consensus_queue.cc:1035] T 3c4bc2f0d8ec4183b4edb54cdbcac1cc P 48789b208dcf4f8ab388470fa43a751b [LEADER]: Connected to new peer: Peer: permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:16:17.133661 16880 raft_consensus.cc:1261] T 3c4bc2f0d8ec4183b4edb54cdbcac1cc P e1a9c39d088a43e7b6f9beee2068deeb [term 1 FOLLOWER]: Refusing update from remote peer 48789b208dcf4f8ab388470fa43a751b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:16:17.133853 17300 consensus_queue.cc:1035] T 76362c2867ba403f9b9011f6b2247137 P e1a9c39d088a43e7b6f9beee2068deeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:16:17.134196 17339 consensus_queue.cc:1035] T 3c4bc2f0d8ec4183b4edb54cdbcac1cc P 48789b208dcf4f8ab388470fa43a751b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9c39d088a43e7b6f9beee2068deeb" member_type: VOTER last_known_addr { host: "127.7.39.130" port: 40479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:16:17.142594 17012 raft_consensus.cc:1261] T 88530fdb379743cf9ab3d1831d4e8d9c P 48789b208dcf4f8ab388470fa43a751b [term 1 FOLLOWER]: Refusing update from remote peer e1a9c39d088a43e7b6f9beee2068deeb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:16:17.143739 17300 consensus_queue.cc:1035] T 88530fdb379743cf9ab3d1831d4e8d9c P e1a9c39d088a43e7b6f9beee2068deeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "48789b208dcf4f8ab388470fa43a751b" member_type: VOTER last_known_addr { host: "127.7.39.131" port: 45011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:16:17.155789 16747 raft_consensus.cc:1261] T 88530fdb379743cf9ab3d1831d4e8d9c P 626e808943c54503bc4bd8cf1cd3e4b5 [term 1 FOLLOWER]: Refusing update from remote peer e1a9c39d088a43e7b6f9beee2068deeb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:16:17.160161 17215 consensus_queue.cc:1035] T 88530fdb379743cf9ab3d1831d4e8d9c P e1a9c39d088a43e7b6f9beee2068deeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "626e808943c54503bc4bd8cf1cd3e4b5" member_type: VOTER last_known_addr { host: "127.7.39.129" port: 43915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 105.027 ms
time per row: 0.0525133 ms
I20240501 14:16:17.206450 7326 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
/tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.39.132:40745
--local_ip_for_outbound_sockets=127.7.39.132
--tserver_master_addrs=127.7.39.190:40389
--webserver_port=41037
--webserver_interface=127.7.39.132
--builtin_ntp_servers=127.7.39.148:39707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 14:16:17.316284 17410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:16:17.316488 17410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:16:17.316550 17410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:16:17.319528 17410 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:16:17.319586 17410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:16:17.319676 17410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.39.132
I20240501 14:16:17.323199 17410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.39.148:39707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.39.132:40745
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.7.39.132
--webserver_port=41037
--tserver_master_addrs=127.7.39.190:40389
--never_fsync=true
--heap_profile_path=/tmp/kudu.17410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.39.132
--log_dir=/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:16:17.323971 17410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:16:17.325037 17410 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:16:17.332926 17418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:16:17.333468 17415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:16:17.334159 17416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:16:17.334787 17410 server_base.cc:1034] running on GCE node
I20240501 14:16:17.335187 17410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:16:17.335697 17410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:16:17.336866 17410 hybrid_clock.cc:648] HybridClock initialized: now 1714572977336821 us; error 58 us; skew 500 ppm
I20240501 14:16:17.338797 17410 webserver.cc:460] Webserver started at http://127.7.39.132:41037/ using document root <none> and password file <none>
I20240501 14:16:17.339377 17410 fs_manager.cc:359] Metadata directory not provided
I20240501 14:16:17.339471 17410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:16:17.342893 17410 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:16:17.344682 17424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:16:17.345655 17410 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 14:16:17.345749 17410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "5852487cc13a4313b0a727f9eb1a6d6a"
format_stamp: "Formatted at 2024-05-01 14:16:11 on dist-test-slave-rnfw"
I20240501 14:16:17.346154 17410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:16:17.364748 17410 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:16:17.365290 17410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:16:17.365435 17410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:16:17.365936 17410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:16:17.366783 17410 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:16:17.366843 17410 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:16:17.366897 17410 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:16:17.366922 17410 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:16:17.374998 17410 rpc_server.cc:307] RPC server started. Bound to: 127.7.39.132:40745
I20240501 14:16:17.375036 17537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.39.132:40745 every 8 connection(s)
I20240501 14:16:17.375952 17410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhUqsdZ/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714572528121239-7326-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 14:16:17.382419 7326 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu as pid 17410
I20240501 14:16:17.383965 17538 heartbeater.cc:347] Connected to a master server at 127.7.39.190:40389
I20240501 14:16:17.384210 17538 heartbeater.cc:464] Registering TS with master...
I20240501 14:16:17.384816 17538 heartbeater.cc:511] Master 127.7.39.190:40389 requested a full tablet report, sending...
I20240501 14:16:17.386034 16600 ts_manager.cc:194] Re-registered known tserver with Master: 5852487cc13a4313b0a727f9eb1a6d6a (127.7.39.132:40745)
I20240501 14:16:17.386775 16600 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.39.132:51069
I20240501 14:16:17.572846 16728 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:16:17.572880 16858 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:16:17.572880 16990 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:16:17.573853 17472 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:16:17.666668 16990 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:16:17.666767 16728 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:16:17.667630 16858 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:16:17.667977 17472 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:16:17.779587 16858 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:16:17.780359 16728 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:16:17.781422 16990 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:16:17.783457 17472 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:16:17.962355 16598 ts_manager.cc:292] Set tserver state for 5852487cc13a4313b0a727f9eb1a6d6a to MAINTENANCE_MODE
I20240501 14:16:18.152123 16990 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:16:18.153141 17472 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:16:18.153950 16728 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:16:18.155843 16858 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:16:18.242399 17472 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:16:18.243188 16990 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:16:18.245940 16858 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:16:18.252688 16728 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:16:18.359952 16728 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:16:18.360536 16990 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:16:18.360559 16858 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:16:18.361095 17472 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:16:18.388593 17538 heartbeater.cc:503] Master 127.7.39.190:40389 was elected leader, sending a full tablet report...
I20240501 14:16:18.442950 16990 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:16:18.442950 17472 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:16:18.444923 16858 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:16:18.448284 16728 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:16:18.482964 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 16666
I20240501 14:16:18.489634 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 16798
I20240501 14:16:18.496044 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 16930
I20240501 14:16:18.502229 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 17410
I20240501 14:16:18.507843 7326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhUqsdZ/build/debug/bin/kudu with pid 16575
2024-05-01T14:16:18Z chronyd exiting
[ OK ] LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer/idx1_rf3_t3_l_3_1 (7799 ms)
[----------] 2 tests from LocationAwareRebalancingParamTest (22191 ms total)
[----------] Global test environment tear-down
[==========] 7 tests from 6 test suites ran. (450405 ms total)
[ PASSED ] 6 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/0, where GetParam() = 4-byte object <00-00 00-00>
1 FAILED TEST